Open Ben621 opened 4 weeks ago
vmaurin
commented
4 weeks ago Hi @Ben621
My company is using aiokafka against AWS MSK 3.6.0 without issue. I think the code you highlight is just happening when you force the api_version to something different to "auto"
Ben621
commented
4 weeks ago Hey @vmaurin
Can please try to print the api_version
In my case we are using 3.4.x,
But I'm unable to connect to get the api_version
But suddenly that connect, I don't know why !!
After. Investigation that work when I set api_version to 2.8.1
Maybe are you familiar with this issue ?
Also I have checked in the GitHub action they are testing client "0.9.0.1" To "2.8.1"
vmaurin
commented
3 weeks ago From the codebase, it sounds parse_kafka_version you pointed out is only use when initializing a producer or a consumer with an api_version. If you let it unset like we do (auto), this method is never called. Then when the client (aiokafka) is connecting to the broker (3.6.0), they will negotiate API version for each individual methods part of the kafka protocol.
I guess then, this parameter is to force a specific version of the protocol, but it should not be used, as Kafka protocol is build for backward and forward compatibility (even today version of the broker probably still support clients in 0.11). So my advice is to let this parameter unset
Ben621
commented
3 weeks ago Hey @vmaurin
can you please check what is you api_version when you set it auto ?
In my case after before they upgrade from kafka 2.7 to kafka 3.5 i was able to connect with auto
but after i got this error
Unable to request metadata from "kafkaserver:9093":suddenly that work once after waiting 2 minutes for connecting and the api_version was 2.6.0
So, when i set api_version= "2.6.0" that connect directly, otherwise got error, or sometime i'm able to connect without knowing why it's happen !!!
@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0
ods
commented
3 weeks ago Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.
ods
commented
3 weeks ago There is an error reported in aiokafka/util.py that restricts compatibility.
Could you please provide a complete report with the traceback?
@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0
The method you're referring to looks weird to me. In my opinion, we should rather drop support for brokers <0.10.0 and rely solely on what ApiVersions returns.
wbarnha
commented
2 weeks ago Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.
I've been meaning to do this in https://github.com/wbarnha/kafka-python-ng for newer Kafka broker versions. If I get a working solution, I'll open a PR here as well.
The current implementation of the aiokafka library within the Confluent platform only supports client versions under 3.0 for Python. There is an error reported in aiokafka/util.py that restricts compatibility.
Additionally, the Confluent platform's version interoperability matrix highlights this limitation.
We request extending compatibility to include higher client versions (e.g., 3.0 or above). This would allow users to leverage the latest features and bug fixes offered by newer aiokafka client releases.