Test kafka with replication_factor 3

[rukai]

This was a missing test case that upon implementation revealed a bug in shotover. Since we cannot create a topic with replication_factor 3 on a kafka cluster with only 1 node, cluster_test_suite is introduced for the cluster tests to use without regressing the single/passthrough tests.

To get the tests to pass we required:

• one change to the test logic - we need to set produce acks to "all"
  • It kind of makes sense in that, adding replicas means that there is now a difference in behavior between acks=1 and acks=all, acks=1 will not wait for the replicas to acknowledge the produced record.
  • On the other hand I'm not entirely sure why we need it since an ack from the leader should be sufficient as the consumer and producer only routes to the leader.
• one change to shotover routing logic - we need to route fetch requests to the leader instead of the replicas.
  • The comment I added to the code justifies the change pretty well. But basically implementing shotover to route fetch requests to the replicas was a mistake that sounded correct at the time. kafka does not allow routing to replicas in the same way that cassandra and redis do. In kafka, replicas are purely for "high availability" with some extra request handling tacked on.

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #1738 will not alter performance

_{Comparing rukai:kafka_rf3_test (62bac6a) with main (57a1362)}

Summary

✅ 39 untouched benchmarks