search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.13k stars 835 forks source link

Infinite retry loop in `gcp_pubsub` output when used with `kafka` input and non-string message keys #1042

Open torfjor opened 2 years ago

torfjor commented 2 years ago

Upstream issue: https://github.com/googleapis/google-cloud-go/issues/5268 Related: https://github.com/googleapis/google-cloud-go/issues/2443

This basically breaks the gcp_pubsub output for anyone consuming from a kafka input where the message keys are not strings.

Attaching a processor that removes the kafka_key metadata solves the issue temporarily until we have a fix:

pipeline:
  processors:
    - bloblang: |
        meta kafka_key = deleted()
Jeffail commented 2 years ago

Not ideal but I've added a troubleshooting section for now given how vague the error is: https://github.com/Jeffail/benthos/commit/30489a57f3a51d31d530e1361661df699a86807c