Kafka input on Benthos 4.22 is incompatible with Event Hubs / Kafka 1.0.0 due to Sarama 1.41.0+

[mfamador]

Starting from Sarama version 1.41.0, every time a consumer group starts or leaves from an Azure Event Hub partition, all running consumers from the same consumer group re-consumes all messages. It appears that the offsets are reset and not committed again. This issue seems to be specific to Azure Event Hubs (Kafka version 1.0.0), and it has also been reproduced, not so often though, using Kafka 1.0.0 docker. I could also reproduce it using plain Golang, so very likely a Sarama's issue, just creating this one here as an heads up for who's using Event Hubs with kafka_input not upgrading to Benthos 4.22 : see https://github.com/IBM/sarama/issues/2677

Reverting back to Sarama 1.40.1 everything works again as expected.

From their breaking changes heads up on 1.41.0 release (https://github.com/IBM/sarama/releases/tag/v1.41.0), we're already pinning the version with target_version to 1.0.0, it should be ok.

input:
  kafka:
    addresses:
    - fooeventhub.servicebus.windows.net:9093
    tls:
      enabled: true
    sasl:
      mechanism: PLAIN
      user: $ConnectionString
      password: 'Endpoint=<READCTED>;EntityPath=test'
    target_version: 1.0.0
    topics:
    - test
    consumer_group: foo

[mfamador]

Considering the new version (1.41.0+) of sarama is breaking Kafka 1.0.0/EventHubs consumers, do you think we can revert back to sarama 1.40.1 ? (https://github.com/benthosdev/benthos/pull/2186)

[Jeffail]

Thanks for the investigation @mfamador!

[mfamador]

There is already a fix on Sarama, we should be able to upgrade again as soon as they release a new version. https://github.com/IBM/sarama/pull/2700

[dnwe]

Fix released in Version 1.42.0