Closed rukai closed 3 months ago
Comparing rukai:kafka_local_shotover_broker_id
(1307c6a) with main
(500f235)
⚡ 1
improvements
✅ 36
untouched benchmarks
Benchmark | main |
rukai:kafka_local_shotover_broker_id |
Change | |
---|---|---|---|---|
⚡ | encode_system.local_result_v5_no_compression |
491.1 µs | 442.8 µs | +10.92% |
progress towards: https://github.com/shotover/shotover-proxy/issues/1526
This PR introduces the
local_shotover_broker_id
field to KafkaSinkCluster. This is the first step towards rack aware routing. In order for shotover to prioritize that requests are sent to kafka brokers within the same rack as shotover itself, shotover must know which rack it is in. This is achieved via thelocal_shotover_broker_id
field which points to a field inshotover_nodes
and declares that node to be the node that this shotover instance is.https://github.com/shotover/shotover-proxy/blob/bc7b2354397067407093627db29c05c5472f1a18/shotover-proxy/tests/test-configs/kafka/cluster-2-racks/topology-rack1.yaml#L7-L17
We could alternatively have a
local_shotover_rack
field which just directly declares what rack shotover is in. But I think thatlocal_shotover_broker_id
would be better since it matches what we do for cassandra and it is also backwards compatible in case we find out that shotover needs to know its broker ID for anything.Since
local_shotover_broker_id
is mandatory I have added it to all integration test topology.yaml's that useKafkaSinkCluster
.We could possibly make the
local_shotover_broker_id
field optional, if its not provided then rack aware routing is not performed. But we have no need for that functionality and its a backwards compatible change to make it optional. So I have not implemented such functionality.Addtional changes
This PR also cleans up the transforms.md documentation as I noticed a bunch of spelling and copy/paste errors.