I love flask_kafka but I have been battling trying to improve the consumer message throughput from the ~30-40 messages per second that I can do on my modern computer.
I tried changing the underlying library from the pure python Kafka library to the librdkafka based confluent_kafka and the message throughput went up from 30-5000 per second with no other changes.
Do you think this is something we can adopt into the mainline flask_kafka project? It's only really the change in configuration, e.g. bootstrap-servers becomes bootstrap.servers that means the change will impact most of the users. :-(
I am not happy with the interrupt behaviour - the consumer sometimes fails to stop when you send the signal with ctrl-c, but perhaps someone with a little more experience can improve this.
Hi,
I love flask_kafka but I have been battling trying to improve the consumer message throughput from the ~30-40 messages per second that I can do on my modern computer.
I tried changing the underlying library from the pure python Kafka library to the librdkafka based confluent_kafka and the message throughput went up from 30-5000 per second with no other changes.
I made a fork and changed the example file to show what change is needed in this library to make it work: https://github.com/andydavidson/flask_kafka
Do you think this is something we can adopt into the mainline flask_kafka project? It's only really the change in configuration, e.g. bootstrap-servers becomes bootstrap.servers that means the change will impact most of the users. :-(
I am not happy with the interrupt behaviour - the consumer sometimes fails to stop when you send the signal with ctrl-c, but perhaps someone with a little more experience can improve this.