Chaos tests and failure injection

[dswang]

• [ ] Revive existing tests in this area #3893 - @rystsov
• [ ] Automate the existing chaos stuff we do - @rystsov
• [ ] Failure points in core code w/ test runner that can exercise permutations of errors, locations.
• [ ] Automate punisher - @mmaslankaprv
• [ ] run chaos tests against our Cloud i.e. chaos monkey - @mmaslankaprv
• [ ] Network or Node-level failure tests - @LenaAn - could use use iptables and or netem - read a cool cockroachDB article on using docker and Linux traffic control to introduce delay etc between nodes, which all run on same host in containers
• [ ] Metrics/observability/tooling for load/chaos testing - use structural logging for network messages and feed them to a state machine to validate for anomalies - @rystsov
• [ ] I'm not exactly sure what specifically load/chaos testing should look like, but it's clear that we are going to need Tooling provisioning environments and automating tests It’s not clear ducktape is what we want here, but maybe Measuring success (crashes are easy, high latency is harder)

JIRA Link: CORE-757

[LenaAn]

@dswang can you please post a link to the cockroachDB article? Couldn't find it after a quick google search

[dswang]

@dswang can you please post a link to the cockroachDB article? Couldn't find it after a quick google search

@ajfabbri knows about this article, this suggestion was from him.

[LenaAn]

Current status:

• I'm not actively working on that, but @rystsov is, here's a repo https://github.com/vectorizedio/chaos
• @ivotron to activate tests from the repo in buildkite and then we can mark first issue as done
• there are some ducktape chaos tests in https://github.com/vectorizedio/redpanda/tree/dev/tests/rptest, but they are obsolete, need to move some last bits from ducktape to the new repo, then we can delete ducktape chaos tests and mark the second issue as done
  • other issues should not be blocked, I'm not aware any work in progress