search

confluentinc / kafka-connect-elasticsearch

Kafka Connect Elasticsearch connector
Other
15 stars 436 forks source link

Capture Kafka key without using it as ID #704

Open kmcclellan opened 1 year ago

kmcclellan commented 1 year ago

I want to use the topic/partition/offset as the ES document ID while still capturing the Kafka message key for audit/search purposes. Unless I'm mistaken, there doesn't appear to be a way to do this with this connector.

kmcclellan commented 1 year ago

Ideally, all Kafka message metadata should be captured in a way that is searchable: topic, partition, offset, timestamp, headers

kmcclellan commented 5 months ago

A year later and there is still no way to capture/search the most basic message metadata within ES. The only reason I wanted Kafka messages copied to ES was to search it. IMO this library is pretty much worthless.

Kafka message keys are used for partitioning and are not intended to be unique. The fact that the implementors of this connector thought they should be used as an Elasticsearch document ID shows a pretty poor understanding of both domains.