search

shotover / shotover-proxy

L7 data-layer proxy
https://docs.shotover.io
Apache License 2.0
87 stars 18 forks source link

KafkaSinkCluster: route DescribeLogDirs request #1823

Closed rukai closed 1 week ago

rukai commented 1 week ago

This PR implements routing for DescribeLogDirs.

When the java driver routes DescribeLogDirs it sends it to a broker specified by the user. When kafka-log-dirs.sh routes DescribeLogDirs it sends an identical request to every node in the cluster. To handle this type of broadcast request shotover routes the request to all nodes in the rack. For other examples look at ListGroups, ListTransactions etc.

We route the request to all nodes in shotover's rack and then combine the results. We alter the log dir path to avoid collisions with results from other racks, the new path looks like actual-kafka-broker-id3:/original/log/dir/path. For more details on the reasoning behind this, look at the comment on fn prepend_destination_to_log_dir.

codspeed-hq[bot] commented 1 week ago

CodSpeed Performance Report

Merging #1823 will not alter performance

Comparing rukai:describe_log_dirs (ced406b) with main (3f76b54)

Summary

✅ 38 untouched benchmarks