search

linkedin / venice

Venice, Derived Data Platform for Planet-Scale Workloads.
https://venicedb.org
BSD 2-Clause "Simplified" License
487 stars 84 forks source link

[controller][server][dvc][vpj][producer][consumer] Bumped Kafka to 3.0 #1237

Closed FelixGV closed 3 weeks ago

FelixGV commented 3 weeks ago

Bumped Kafka from 2.4.1.65 to 3.0.1.76.

Addressed incompatible API changes affecting tests.

How was this PR tested?

Existing suite.

Also wrote a unit test for Kafka's NetworkClient::leastLoadedNode API, and ran it against an older and newer version of Kafka to confirm the change in behavior we care for, but I did not check in this test as we usually don't want to unit test dependencies, plus this class seems to change a lot across versions, which would make the test tedious to maintain.

Does this PR introduce any user-facing changes?

FelixGV commented 3 weeks ago

Looks like 1280 (at least) has a regression... will need to check.

FelixGV commented 3 weeks ago

There are incompatible semantic changes in the Kafka admin API which I started debugging but did not finish... for now, I suggest bumping within the 2.4 line rather than going to 3. I have opened https://github.com/linkedin/venice/pull/1238 to do this.