scylladb / kafka-connect-scylladb

Kafka Connect Scylladb Sink
Apache License 2.0
46 stars 22 forks source link

Is it possible to use dynamic keyspace? #47

Open tunix opened 3 years ago

tunix commented 3 years ago

Hi,

I've a multi-tenant application. I want to read messages from a single Kafka topic and split messages across different scylladb keyspaces based on a field inside the Kafka message.

Does the connector support this? If not, would you consider supporting it?

avelanarius commented 3 years ago

Currently, it is not supported.

As a workaround, assuming the number of destination keyspaces is small and fixed, you could launch N instances of Sink Connector (each one writing to a specific keyspace) and use a Filter (Confluent version) SMT transformation on each connector instance to filter the messages to a correct keyspace.

We will consider adding this feature in the future.