Add headers support to the redpanda_migrator_bundle output

redpanda-data / connect

Fancy stream processing made operationally mundane

https://docs.redpanda.com/redpanda-connect/about/

8.14k stars 840 forks source link

Add headers support to the redpanda_migrator_bundle output #2984

Closed mihaitodor closed 2 weeks ago

mihaitodor commented 2 weeks ago

This is a quick hack which we can use, but I'd rather have a more robust fix. Also, this approach is problematic if users have custom headers which are prefixed with kafka_.

Would you consider emitting Kafka headers from the input as an array using structured metadata and maybe call the field kafka_header? This would be a breaking change for existing components, but maybe we can just do it for as part of FranzRecordToMessageV1() in internal/impl/kafka/franz_reader.go in #2918.

rockwotj commented 2 weeks ago

This seems better as a quick fix - but yea it would be better to have some way to be more robust about this for the migrator as you mention in the PR description. Maybe can happen after Ash's Kafka refactor PR