search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.46k stars 3.7k forks source link

Custom Json extractors for Kafka lookup module #8097

Closed satybald closed 4 years ago

satybald commented 5 years ago

Description Extend Kafka lookup functionality to use Kafka JSON payloads to extract key/values pair and do simple Jq expression transformation.

Motivation Kafka log compacted topics are a perfect feature for a lookup data source(aka facts tables). However, the current implementation of Kafka Lookups uses only topic message payload as values and topic key as the key for druid lookup map. This a rigid setup, requires a team to transform existing topics to plain key/value topic.

For teams using Kafka as a data source and druid as an analytics engine, it's a nessary feature to extract values and keys from the message payload as Global Cached Lookups.

stale[bot] commented 4 years ago

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.