search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.89k stars 3.78k forks source link

roachtest: run tests against larger clusters #26978

Closed tbg closed 5 years ago

tbg commented 6 years ago

A user has privately shared multiple issues found working with generally ~32 node clusters, including stuck requests during IMPORT and minutes of failed liveness heartbeats (the first likely and the second definitely related to a GC'able replica that isn't getting removed in due time).

While we generally have some theory and understanding about these issues, we should be more proactive about running into them ourselves. A few tests that run heavy rebalancing plus chaos along with regular activity (that is somewhat latency-sensitive, i.e. we would notice if it got stuck for 30+s) on most ranges should be sufficient.

tbg commented 5 years ago

We're running various imports/restores against clusters of size 8 and 32, and that has turned up a number of issues that are now fixed (#31409). There's always more to do, but this issue is too general.