googleapis / java-pubsub-group-kafka-connector

Apache License 2.0
41 stars 18 forks source link

Missing ServiceLoader manifest (for Kafka >= 3.6) #310

Open mkuthan opened 4 months ago

mkuthan commented 4 months ago

Kafka Connect has added new plugin discovery with ServiceLoader since 3.6. If the plugin doesn't register itself using ServiceLoader KafkaConnect reports:

One or more plugins are missing ServiceLoader manifests may not be usable with plugin.discovery=service_load: [
file:/opt/plugins/pubsub-group-kafka-connector-1.2.0.jar    com.google.pubsub.kafka.sink.CloudPubSubSinkConnector   sink    3.7.0
file:/opt/plugins/pubsub-group-kafka-connector-1.2.0.jar    com.google.pubsub.kafka.source.CloudPubSubSourceConnector   source  3.7.0
file:/opt/plugins/pubsub-group-kafka-connector-1.2.0.jar    com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector    sink    3.7.0
file:/opt/plugins/pubsub-group-kafka-connector-1.2.0.jar    com.google.pubsublite.kafka.source.PubSubLiteSourceConnector    source  3.7.0
Read the documentation at https://kafka.apache.org/documentation.html#connect_plugindiscovery for instructions on migrating your plugins to take advantage of the performance improvements of service_load mode. To silence this warning, set plugin.discovery=only_scan in the worker config.

I would like to set plugin.discovery to service_load to speed up the cluster startup. But it requires proper plugin registration with ServiceLoader.

Now it's just a WARN but it could change in the future version of Kafka.