[QUESTION] Does aiokafka can replace the unmaintained kafka-python project?

[LiorA1]

Hi,

I'm new to Kafka and trying to get an initial sense of the available libraries & theirs status.

Does 'aiokafka' still based on top of the unmaintained kafka-python project?

P.S- Maybe its worth to open a "Discussions" tab ?

[Lameorc]

Seconding the above, I'll add some more context: The kafka-python library has several discussions as to whether it's maintained or when new release is happening:

• https://github.com/dpkp/kafka-python/issues/2349
• https://github.com/dpkp/kafka-python/issues/2290
• https://github.com/dpkp/kafka-python/issues/2353

The main reason this is of interest to me is that the kafka python doesn't support the latest kafka versions and I don't expect it will based on the above discussions. I've not managed to find any sort of compatibility matrix but we're trying to switch to upgrade kafka v3+ which seems impossible with this library as of now?

[vmaurin]

I am not 100% sure as I haven't checked actively kafka release notes since v2.8.x, but Kafka protocol is known to be forward and backward compatible since v0.8.x

Part of client connection, there is a phase of protocol supports, so a client lib with the 0.8.x protocol should be able to talk to a broker with 3+. Then it might not be always the optimal way, and you will lack new protocol features

[Lameorc]

Huh, shame on me for not double checking things I've been told... I'll try and integrate my app with newer kafka and see how that goes. Thanks for the hint!

[jainal09]

So, is there a roadmap ahead for aio-kafka. What is the plan for supporting future kafka releases if kafka-python is not maintaining there source. Also, are there plans to replace kafka-python with confluent-kafka-python library which is well maintained and up-to date

[marcosschroh]

Hi all,

We have migrated our kafka cluster to the latest kafka versión and everything kept working. We run some tests against kafka-python/aiokafka and we did not find any issues. It is true that the new kafka version indicates a breaking change but the API remains the same. Anyway, it is a bit worrying that aiokafka depends on kafka-python (unmaintained)

[ods]

There is a plan to remove dependency on kafka-python. But the main problem is that we need more maintainers for aiokafka project.

[LiorA1]

in favor of confluent-kafka-python ? Or to be a stand-alone ?

to open a tab of required tickets somehow cant motivate people pick them up?

[wbarnha]

Hi folks,

As of recent events, I have received push access for the existing kafka-python project. I'm aware of aiokafka's current dependency of kafka-python, and I have received access from the original maintainers to revive the kafka-python project. The current plan is to migrate the CI/CD pipeline for kakfa-python to GitHub, and then immediately create a release of the current state of the project. After that, I'll review existing PRs for kafka-python, and hopefully eliminate the need for us to worry about our kafka-python dependency.

Re: confluent-kafka-python

Sadly, this project is an entirely different beast when it comes to asynchronous functionality. There's still some work missing from the project and it's not suitable to completely replace aiokafka. https://github.com/confluentinc/confluent-kafka-python/pull/1448 for example has been pending merge for quite some time and most of their library is used in synchronous contexts. I'm not sure what their intended direction for that project is, but until there's obvious asynchronous support, I'm trying to avoid it. I've (unsuccessfully) performed various experiments with confluent-kafka-python in asynchronous contexts inside https://github.com/faust-streaming/faust/pull/418 and I'm not particularly fond of Python's Kafka projects to be entirely dependent on Confluent's library. The librdkafka bindings built for the project are another beast that can be difficult to troubleshoot. I'd go as far as to argue that the current aiokafka + kafka-python implementation in pure Python/Cython has some benefits offered in typing and compatibility with other Python implementations such as PyPy.

I understand the perils of relying on FOSS maintainers, many people such as myself are trying to balance their jobs and personal lives and are inevitably fated to depart FOSS support at some point due to circumstances beyohd their control. I do think that hat we should look into moving towards a community that has more say in voting on the direction of aiokafka and handling pending PRs.

I would be willing to step up to the plate to support aiokafka, but I'm already burdened by Faust and Kafka-Python, I don't want to neglect other projects.

[wbarnha]

Wrapping up a workflow for kafka-python in https://github.com/dpkp/kafka-python/pull/2378, hopefully this will be the change that brings the project back to life!

[ods]

in favor of confluent-kafka-python ? Or to be a stand-alone ?

Standalone. The plan is to incorporate the code aiokafka depends on.