jepsen: make the Jepsen tests first-class citizens

[knz]

Make the Jepsen tests a first-class citizen

• [ ] create a dashboard linked/embedded in the main web site that displays the status from the latest tests with the date/time of the last test run #19695
• [ ] deploy separate Jepsen testing for each released CockroachDB version concurrently

Better automate the tests

• [x] make the tests use pre-allocated VMs instead of terraforming on each run.
• [ ] upload the perf statistics gathered by Jepsen to a separate repo so we can see how a test performance changes over time. #19696
• [ ] use different running times for different tests and different nemesis combinations. #19697

Ease the troubleshooting process

• [ ] make it simpler to re-launch a failed test using the exact same configuration (e.g. a single link to download a script that does the same thing).

Fix bugs/limitations in the test framework

• [ ] make nemeses and test initialization less flaky by using cockroach quit more, use signals less, take inspiration from the CLI interactive tests.
• [x] jepsen: failure due to unexpected retry error in Sets test #15733
• [x] jepsen: the Register test is improperly configured #15734
• [x] jepsen: the start-stop nemesis is flaky/incorrect #15736
• [x] jepsen: the Sequential test is probably incorrect #15737
• [x] triage remaining issues in #14171
• [ ] jepsen: (re-)enable the big-skews nemesis #15893
• [ ] enable the split nemesis (and maybe others?) #19697

Extend the test suite

• [ ] tests: possible linearizability anomaly when running linearizable #5517
• [ ] add testing using a nemesis that introduces data corruption or disk errors #19656
• [x] run for longer than 3m per test (#17815 needs several hours to reproduce)

[knz]

cc @bdarnell @spencerkimball

[bdarnell]

jepsen-io/jepsen#178 fixes the register tests and reduces some other flakiness. The sequential tests are still failing 100% of the time, and the most common failure in the other tests is #16450.

[bdarnell]

The jepsen tests are in pretty good shape these days. I think that aside from the automatic filing of issues (#19694), they're already "first-class citizens". I've added separate issues for most of the remaining points here, so I'm closing this meta-issue. Feel free to open new issues for the points abandoned here if you think they're worth keeping.