confluentinc / kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases
Other
22 stars 958 forks source link

Add/remove connector restarts all connectors #405

Open richard6226 opened 6 years ago

richard6226 commented 6 years ago

I'm using kafka-connect-jdbc. I'm using a JDBC source connector to load a number of large tables from a Microsoft SQL database into Kafka topics which I then use a HBase sink connector to persist in HBase tables. So I use bulk mode for the initial load of the tables and then I'm planning on using timestamp incremental connector once the initial load of Hbase tables is done.

Using distributed mode it appears that each time I add a new connector (to load another table) all other connectors are stopped and start over. Essentially this restarts my other bulk connectors. Is this normal operation? Is there a way to add a new connector without restarting all other connectors?

jeffbeagley commented 5 years ago

I see this reference to config.action.reload within the source code, but am not sure I'm quite following its description

https://github.com/axbaretto/kafka/blob/16b7bab62f25fd82a2287ba7b5abca44d85efba9/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L98

            "The action that Connect should take on the connector when changes in external " +
            "configuration providers result in a change in the connector's configuration properties. " +
            "A value of 'none' indicates that Connect will do nothing. " +
            "A value of 'restart' indicates that Connect should restart/reload the connector with the " +
            "updated configuration properties." +
            "The restart may actually be scheduled in the future if the external configuration provider " +
            "indicates that a configuration value will expire in the future.";