search

wbarnha / kafka-python-ng

Fork for Python client for Apache Kafka
https://wbarnha.github.io/kafka-python-ng/
Apache License 2.0
80 stars 11 forks source link

Twisted interface #126

Open wbarnha opened 8 months ago

wbarnha commented 8 months ago

Kafka-Python claims to be "async" but this seems to just mean "uses lots of threads to put your I/O into the background" and not actually "event-driven". It would be nice to have an interface which avoids the overhead (thread stacks, GIL switching, etc) and risks (sharing too much state, deadlocks, difficult to debug differing behavior at scale) of threads, but presents a similar interface.

I filed the ticket with the summary "twisted" because of my particular biases, but this should be done in a way which is as framework-neutral as possible, since supporting the asyncio stdlib module at some point in the future would probably be interesting as well, not to mention Tornado and Vanilla and so on.

Given that one would be consuming a queue, perhaps this would be a good landmark project for https://github.com/twisted/tubes which, despite its github URL, has an (almost) framework-neutral architecture and lots of good abstractions around flow control and propagating backpressure.

wbarnha commented 8 months ago

This was already achieved by https://github.com/ciena/afkak but sadly their library has not been maintained well.