Our tests currently use a rather static Kafka/Redpanda cluster and test the happy path. We should also have a test that deals with a messy cluster.
For that I think we can have the following long-running test job:
have cluster with multiple nodes (e.g. 3)
have one producer and one consumer and a continuous flow of data
have a chaos monkey killing nodes or re-scaling the cluster (incl. re-balancing)
This is a bit related to #49 which captures the unit-test aspects of our retry logic. Also relates to #147 which would probably have found by proper chaos testing.
To set expectations, once this testing is in place I predict a non trivial amount of time will be required to diagnose / debug and / fix whatever it turns up
Our tests currently use a rather static Kafka/Redpanda cluster and test the happy path. We should also have a test that deals with a messy cluster.
For that I think we can have the following long-running test job:
This is a bit related to #49 which captures the unit-test aspects of our retry logic. Also relates to #147 which would probably have found by proper chaos testing.