search

bytewax / bytewax

Python Stream Processing
https://docs.bytewax.io/
Apache License 2.0
1.46k stars 59 forks source link

[FEATURE] Provide the option to include Kafka headers with Kafka InputConfig #140

Closed PiePra closed 8 months ago

PiePra commented 1 year ago

Is your feature request related to a problem? Please describe. Currently it is not possible to parse Kafka headers using the Kafka input config. Kafka headers might contain valuable information. E.g. CloudEvents propagate various fields like event_id and timestamp in kafka headers.

Describe the solution you'd like I would like to be able to optionally retrieve Kafka Headers alongside with the message to use the header information in my dataflow.

Describe alternatives you've considered The alternative is to customize the message to include all necessary information in the body

davidselassie commented 8 months ago

https://github.com/bytewax/bytewax/pull/332 added support for this and has been released in https://github.com/bytewax/bytewax/releases/tag/v0.18.0 .