confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
113 stars 1.04k forks source link

Mount (Volumes) not working when embedding Kafka connect into Ksqldb-server #7833

Open bluedog13 opened 3 years ago

bluedog13 commented 3 years ago

Describe the bug The "volumes" mapping does not seem to work when trying to embed Kafka connector in Ksqldb Server

To Reproduce Below is my docker file

ksqldb-server:
        image: confluentinc/ksqldb-server:0.18.0
        hostname: ksqldb-server
        container_name: ksqldb-server
        depends_on:
          - broker
          - schema-registry
        ports:
          - "8088:8088"
        volumes:
          - "/usr/local/share/kafka/connectors/debezium-debezium-connector-mysql/lib:/usr/share/kafka/plugins/"
        environment:
          KSQL_LISTENERS: "http://0.0.0.0:8088"
          KSQL_BOOTSTRAP_SERVERS: "broker:9092"
          KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
          KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
          KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
          # Configuration to embed Kafka Connect support.
          KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster"
          KSQL_CONNECT_BOOTSTRAP_SERVERS: "broker:9092"
          KSQL_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
          KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
          KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
          KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "_ksql-connect-configs"
          KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "_ksql-connect-offsets"
          KSQL_CONNECT_STATUS_STORAGE_TOPIC: "_ksql-connect-statuses"
          KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
          KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
          KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
          KSQL_CONNECT_PLUGIN_PATH: "/usr/share/kafka/plugins"

Expected behavior Kafka connect should work using the required plugin/connector

Actual behaviour { "error_code" : 500, "message" : "Failed to find any class that implements Connector and which name matches io.debezium.connector.mysql.MySqlConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='7.0.0-ccs-beta210423230724', encodedVersion=7.0.0-ccs-beta210423230724, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='7.0.0-ccs-beta210423230724', encodedVersion=7.0.0-ccs-beta210423230724, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='7.0.0-ccs-beta210423230724', encodedVersion=7.0.0-ccs-beta210423230724, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='7.0.0-ccs-beta210423230724', encodedVersion=7.0.0-ccs-beta210423230724, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='7.0.0-ccs-beta210423230724', encodedVersion=7.0.0-ccs-beta210423230724, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='7.0.0-ccs-beta210423230724', encodedVersion=7.0.0-ccs-beta210423230724, type=source, typeName='source', location='classpath'}" }

Additional context I have the required jar files installed locally on my machine. Below are the jar files in the folder (usr/local/share/kafka/connectors/debezium-debezium-connector-mysql/lib)

  1. antlr4-runtime-4.7.2.jar
  2. debezium-connector-mysql-1.1.0.Final.jar
  3. debezium-ddl-parser-1.1.0.Final.jar
  4. mysql-connector-java-8.0.16.jar
  5. debezium-api-1.1.0.Final.jar
  6. debezium-core-1.1.0.Final.jar
  7. mysql-binlog-connector-java-0.19.1.jar
bluedog13 commented 3 years ago

Also running the below command result in "Resource not found" (local port 8088 maps to 8088 in the docker file) curl http://localhost:8088/connectors

Resource not found