search

thatdot / quine

Quine • a streaming graph • https://quine.io • Discord: https://discord.gg/GMhd8TE4MR
https://quine.io
Other
295 stars 39 forks source link

Support for extracting data from the key or headers of a kafka message #31

Open olivierdeckers opened 1 year ago

olivierdeckers commented 1 year ago

Currently, the payload is the only part of a kafka record that’s parsed from what I can tell. It would be nice to be able to extract information from headers and keys as well.

emanb29 commented 1 year ago

This is a great suggestion. We don't currently have a plan to add such a feature, but this would be a great contribution.

Some initial thoughts to contribute: To preserve API compatibility, we could add a new (optional / opt-in) flag to a kafka ingest configuration like includeMeta: Boolean = false (indicating it defaults to false if unspecified in the JSON payload)

When includeMeta is true, the $that value passed to the ingest query could take a different shape -- for example

{
  kafkaMeta: ...,
  data: ... (i.e., $that if includeMeta were false)
}

Negotiating this type change between the "ingest source layer" (i.e., "kafka") and the "ingest format layer" (e.g., CypherJson, CypherProtobuf) could be a bit tricky -- the format needs to be aware since that layer goes bytes => intermediate records but the ingest source layer also needs to be aware since that layer goes connection to kafka => record bytes. Not sure how to resolve that.