confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
89 stars 1.04k forks source link

Support CREATE OR REPLACE for connectors #6837

Open rmoff opened 3 years ago

rmoff commented 3 years ago

Feature request: support CREATE OR REPLACE <SINK|SOURCE> CONNECTOR

This would be in line with the CREATE OR REPLACE syntax being added elsewhere in ksqlDB.

Kafka Connect's REST API supports create/update of connectors using the PUT operation, making it easy to script idempotent connector creation / updates to a required state. It would be great if we could do this in ksqlDB too.

6067 added a workaround, namely

DROP CONNECTOR IF EXISTS <name>;
CREATE SINK CONNECTOR <name> [………]

but this is (a) less elegant and (b) more overhead (two operations instead of one).

The reason the existing CREATE CONNECTOR IF NOT EXISTS doesn't suffice here is that it wouldn't handle updating an existing connector to a new configuration.

/cc @agavra :)

abraham-leal commented 2 years ago

+1, hitting this issue constantly when trying to manage connectors with ksqlDB. IMO, this makes it an incomplete implementation, since DROPing and then CREATEing a connector isn't the same as using the PUT capabilities of connector updates in Kafka Connect.

yoniwg commented 5 months ago

+1. For sink connector this is even more serious, since the DROP operation deletes the consumer group and offsets are reset.