search

scylladb / kafka-connect-scylladb

Kafka Connect Scylladb Sink
Apache License 2.0
42 stars 22 forks source link

Offset Storage Table does not support multiple connectors for the same topic #62

Open hartmut-co-uk opened 2 years ago

hartmut-co-uk commented 2 years ago

Precondition

scylladb.offset.storage.table.enable=true (default)

Description

As a developer I'd like to be able to have multiple connectors consuming the same topic. Currently this is not possible, since the scylla table created and used for tracking offsets kafka_connect_offsets (default) has a PK of (topic, partition) and therefore only supports tracking offset for a single connector.

Proposed Solution

Add ad additional column connector_name to the tracking table + PK...

CREATE TABLE poc.kafka_connect_offsets (
    connector_name text,
    topic text,
    partition int,
    offset bigint,
    PRIMARY KEY ((connector_name, topic, partition))
)
hartmut-co-uk commented 2 years ago

Would a PR be welcome? Would there be any other cols worth adding?

haaawk commented 2 years ago

Yes. Contributions are welcome.

Bouncheck commented 1 year ago

Hi, this can be solved by setting scylladb.offset.storage.table=another_table. This way connectors won't clash with each other if you need to store offsets on Scylla. While not having to modify default name would be a convenient feature, I don't think there currently are plans to implement that.