search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.14k stars 839 forks source link

`kafka` input stalls on high lag #1923

Open robertmarsal opened 1 year ago

robertmarsal commented 1 year ago

Since 4.9.1 (specifically this commit: https://github.com/benthosdev/benthos/commit/349ba19cf45fd137ce8bff8462dac5d1a4388079) when one of our Kafka topics has a high lag (more than 2000 pending messages) Benthos stops consuming without returning any errors and just stalls there.

It seems like the downgrade of the sarama library introduces this bug as previous versions before the rollback can handle this scenario just fine.

Looks like you might be waiting on https://github.com/Shopify/sarama/issues/2358 to be resolved to upgrade but that seems unlikely given how long has past. Would it be possible to upgrade to the latest version? Thanks

robertmarsal commented 1 year ago

We've updated to the latest version of sarama however it doesn't seem to fix this issues consistently so there might be other changes we are missing 🤔

Jeffail commented 1 year ago

Hey @robertmarsal, have you given the kafka_franz input a try? My medium-term plan is to deprecate kafka for it.

robertmarsal commented 1 year ago

Hi @Jeffail we looked into it and the only thing stopping us at the moment is the lack of extract_tracing_map in kafka_franz without it all our traces will get disconnected. Happy to add that if I could get some pointers 🤔

jeongukjae commented 2 months ago

Add a PR to add extract_tracing_map in kakfa_franz :) https://github.com/redpanda-data/connect/pull/2836