Not able to install connector

[Mayurg6832]

I am not able to install the connector I am using Kafka version 3.5.0 and Scala version 2.13 (kafka_2.13-3.5.0) I have installed kafka-connect-iceberg-sink-0.4.0-plugin.zip from the releases My directory structure is like

home
├── connectors
│   ├── kafka-connect-iceberg-sink-0.4.0-plugin.zip
│   └── kafka-connect-iceberg-sink (after unzip)
└── kafka_2.13-3.5.0
    └── config
        └── connect-distributed.properties

and have set plugin path in connect-distributed.properties as plugin.path=home/connectors

When stating kafka-connect I am getting

[2023-09-04 17:43:15,806] INFO Loading plugin from: /home/connectors/kafka-connect-iceberg-sink-0.4.0-plugin.zip (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-09-04 17:43:15,810] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink-0.4.0-plugin.zip} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-09-04 17:43:15,810] INFO Loading plugin from: /home/connectors/kafka-connect-iceberg-sink (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-09-04 17:43:30,734] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)

But the connector is not getting installed checked with curl -s -XGET http://localhost:8083/connector-plugins|jq '.[].class' But getting only

"org.apache.kafka.connect.mirror.MirrorCheckpointConnector"
"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector"
"org.apache.kafka.connect.mirror.MirrorSourceConnector"

I am able to install other connectors like debezium PSQL and confluent JDBC

Kindly help Thanks

[duc-dn]

Hi @Mayurg6832, you should set plugin.path=/home/connectors/kafka-connect-iceberg-sink. I think it will resolve your problem

[Mayurg6832]

Hey @duc-dn, Thanks for the response have tried that as well It registers all jar files inside the folder but still not adding the plugin

....
[2023-09-06 07:07:45,073] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink/twill-discovery-core-0.6.0-incubating.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-09-06 07:07:45,074] INFO Loading plugin from: /home/connectors/kafka-connect-iceberg-sink/tephra-hbase-compat-1.0-0.6.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-09-06 07:07:45,077] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink/tephra-hbase-compat-1.0-0.6.0.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-09-06 07:07:45,077] INFO Loading plugin from: /home/connectors/kafka-connect-iceberg-sink/postgresql-42.3.3.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-09-06 07:07:45,137] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink/postgresql-42.3.3.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-09-06 07:07:45,141] INFO Loading plugin from: /home/connectors/kafka-connect-iceberg-sink/checker-qual-3.5.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-09-06 07:07:45,151] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink/checker-qual-3.5.0.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-09-06 07:07:45,152] INFO Loading plugin from: /home/connectors/kafka-connect-iceberg-sink/kafka-connect-iceberg-sink-0.4.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-09-06 07:07:45,158] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/connectors/kafka-connect-iceberg-sink/kafka-connect-iceberg-sink-0.4.0.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
...

[duc-dn]

@Mayurg6832, if you built services in docker-compose, you can refer to my docker file. This is my kafka-connect docker file and it worked for me.

FROM confluentinc/cp-kafka-connect-base:latest

COPY ./docker-config/getindata_iceberg_sink /data/plugins/getindata_iceberg_sink

ENV CONNECT_PLUGIN_PATH="/usr/share/confluent-hub-components,/usr/share/java,/data/plugins/getindata_iceberg_sink"

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.7.4
RUN confluent-hub install --no-prompt debezium/debezium-connector-mysql:2.2.1
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.7.4
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.6.1
RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:2.2.1

Besides, you should refer to this link https://rmoff.net/2020/06/19/how-to-install-connector-plugins-in-kafka-connect/

[Mayurg6832]

I am not using docker, I have downloaded kafka from https://kafka.apache.org/downloads and running it locally.