search

shotover / shotover-proxy

L7 data-layer proxy
https://docs.shotover.io
Apache License 2.0
86 stars 17 forks source link

Fix Kafka java driver multi shotover tests #1588

Closed rukai closed 5 months ago

rukai commented 6 months ago

This PR brings us closer to full support for the java driver. The java driver integration tests that are currently disabled are now enabled but some of the test cases within those integration tests are still disabled. This is done by calling a minimal_test_suite function and calling it when running the java driver.

The fixes to shotover was to handle prefetching topic metadata for fetches using the newer protocol version used by the java driver. The new protocol specifies topics by their id instead of their name. Previously we were only prefetching topic metadata by name but now we also do it by topic id. This involved:

  1. finding a list of all topic id's in the current message batch. (route_requests method)
  2. Including these topic id's in the metadata request (get_metadata_of_topics method)
    • I chose to revert to a lower api_version when not dealing with topic id's, this isnt required but I figured we may as well support the older kafka versions since it doesn't add much complexity.
codspeed-hq[bot] commented 5 months ago

CodSpeed Performance Report

Merging #1588 will not alter performance

Comparing rukai:fix_kafka_java_multi_shotover_tests (e7ab9c9) with main (e69fdac)

Summary

✅ 37 untouched benchmarks