New Feature:
Adding new connector configuration index.name.override which allows for defining a static index name that documents will be written to.
Current Behavior:
Current logic for the index name is based off of the incoming kafka topic or data stream topic.
Problem this Solves:
There currently isn't a way to change the topic with connector configuration. The only solution is to use the kafka connect transformer "org.apache.kafka.connect.transforms.RegexRouter" to remap the incoming topic(s).
The problem with this solution is when using the key.ignore.id.strategy with value of "topic.partition.offset", the topic in this key is always mapped to the static value defined by the RegexRouter, and will cause document collisions.
Proposed Solution:
Add new configuration index.name.override which allows for defining a static index name that documents will be written to regardless of source topic(s).
New Feature: Adding new connector configuration index.name.override which allows for defining a static index name that documents will be written to.
Current Behavior: Current logic for the index name is based off of the incoming kafka topic or data stream topic.
Problem this Solves: There currently isn't a way to change the topic with connector configuration. The only solution is to use the kafka connect transformer "org.apache.kafka.connect.transforms.RegexRouter" to remap the incoming topic(s).
The problem with this solution is when using the key.ignore.id.strategy with value of "topic.partition.offset", the topic in this key is always mapped to the static value defined by the RegexRouter, and will cause document collisions.
Proposed Solution: Add new configuration index.name.override which allows for defining a static index name that documents will be written to regardless of source topic(s).
Existing Issues: https://github.com/aiven/opensearch-connector-for-apache-kafka/issues/89