search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

jepsen: More nemeses, flexible configuration #19697

Closed bdarnell closed 3 years ago

bdarnell commented 6 years ago

Our jepsen tests support some nemesis configurations that we are not currently using: split, and a family of -skews configurations. We should enable both the split nemesis and at least one of the skews nemeses.

We currently run each (test, nemesis) combination for six minutes, so adding a new nemesis adds a significant amount of runtime to the test suite. The failure noted in #17815 occurs once per several hours of runtime, so increasing the test duration would improve our ability to find failures like this. However, adding configurations or increasing runtime has a cost in machine resources. We should figure out how we want to handle this, which may include running some test+nemesis configurations longer than others. For example, the sets and sequential tests seem to be our most sensitive ones; the other tests have rarely noticed an issue that wasn't also caught by one of those too, so we should give them a larger fraction of our test runtime budget.

github-actions[bot] commented 3 years ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!