search

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
337 stars 72 forks source link

Kafka client collaboration #122

Open crepererum opened 1 year ago

crepererum commented 1 year ago

Hi there,

I see you are using rdkafka for your Kafka interaction. We (= InfluxDB IOx) implemented our own pure-Rust async client called rskafka which you might find useful. There is also an intro blog post: https://www.influxdata.com/blog/building-simple-pure-rust-async-apache-kafka-client/

I think you might be missing a feature or two, but nothing fundamental (to my knowledge).

Let me know if you have questions, or want to chat.

Cheers, Marco

PS: Also feel free to close this ticket.

houqp commented 1 year ago

Thanks for sharing @crepererum ! I think it would be a cool idea to try rskafka to reduce C dependencies :)

rtyler commented 1 year ago

@crepererum I have been looking over rskafka which seems like it might not be suitable for our use until a time when it supports consumer groups and offset tracking.

Based on the readme in the repository I cannot tell if such contributions would be welcome or not.

crepererum commented 1 year ago

Based on the readme in the repository I cannot tell if such contributions would be welcome or not.

They are, it's just that we (InfluxData) won't implement them. We can help w/ reviews though.