We are encountering a performance issue with Kafka Connect plugin loading due to missing ServiceLoader manifests. The logs indicate that the following plugins are missing the necessary ServiceLoader manifests:
The absence of ServiceLoader manifests can significantly impact plugin loading times. Kafka Connect may spend extra time trying to resolve or handle these plugins, which can result in slower startup times and potentially affect overall performance.
Logs:
2024-09-17 06:04:02,497 WARN One or more plugins are missing ServiceLoader manifests may not be usable with plugin.discovery=service_load: [
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.IcebergSinkConnector sink 1.5.2-kc-0.6.19
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.transforms.CopyValue transformation undefined
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.transforms.DebeziumTransform transformation undefined
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.transforms.DmsTransform transformation undefined
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.transforms.JsonToMapTransform transformation undefined
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.transforms.KafkaMetadataTransform transformation undefined
file:/opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ io.tabular.iceberg.connect.transforms.MongoDebeziumTransform transformation undefined
]
Read the documentation at https://kafka.apache.org/documentation.html#connect_plugindiscovery for instructions on migrating your plugins to take advantage of the performance improvements of service_load mode. To silence this warning, set plugin.discovery=only_scan in the worker config. (org.apache.kafka.connect.runtime.isolation.Plugins) [main]
Description:
We are encountering a performance issue with Kafka Connect plugin loading due to missing ServiceLoader manifests. The logs indicate that the following plugins are missing the necessary ServiceLoader manifests:
Plugin Path: /opt/kafka/plugins/iceberg-kafka-connect-runtime-hive-0.6.19/ Connector: io.tabular.iceberg.connect.IcebergSinkConnector (version 1.5.2-kc-0.6.19) Transformations: io.tabular.iceberg.connect.transforms.CopyValue (undefined version) io.tabular.iceberg.connect.transforms.DebeziumTransform (undefined version) io.tabular.iceberg.connect.transforms.DmsTransform (undefined version) io.tabular.iceberg.connect.transforms.JsonToMapTransform (undefined version) io.tabular.iceberg.connect.transforms.KafkaMetadataTransform (undefined version) io.tabular.iceberg.connect.transforms.MongoDebeziumTransform (undefined version)
Impact:
The absence of ServiceLoader manifests can significantly impact plugin loading times. Kafka Connect may spend extra time trying to resolve or handle these plugins, which can result in slower startup times and potentially affect overall performance.
Logs: