search

fede1024 / rust-rdkafka

A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka
MIT License
1.63k stars 278 forks source link

Update docker to use new broker #123

Closed gklijs closed 5 years ago

gklijs commented 5 years ago

We recently updated the librdkafka version to 1.0.0 but did not update the broker for the tests. This is fine for now since it's backwards compatible, but probably will cause trouble when testing new features like the idempotent producer. I updated the docker-compose to use the current zookeeper and kafka from confluent. Tests are running locally now, will make a pr soon.

gklijs commented 5 years ago

https://github.com/fede1024/rust-rdkafka/pull/124 is the pr

gklijs commented 5 years ago

@flavray could you take a look at the pr please and/or let me know if this is useful.

Something else, but don't want to create a separate issue for it. Yesterday I did a talk in witch I compared the two Rust clients with two JVM versions of the same thing. Slides are here https://speakerdeck.com/gklijs/adding-some-rust-to-your-kafka. From this it seems the StreamConsumer is putting unnecessary load on the Kafka by polling too much, when the processing of messages takes longer/there are not that many messages. Do you think it's worth addressing this issue?