search

confluentinc / cp-docker-images

[DEPRECATED] Docker images for Confluent Platform.
Apache License 2.0
1.14k stars 703 forks source link

Docker Kafka-connect does not start. Having issues with onfiguration key.converter #418

Open gmocquet opened 6 years ago

gmocquet commented 6 years ago

Hi,

When I try to start the confluentinc/cp-kafka-connect:latest image, but without success. I got the following sack trace:

===> Launching kafka-connect ... 
dev-kafka-connect        | Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value "io.confluent.connect.avro.AvroConverter" for configuration key.converter: Class "io.confluent.connect.avro.AvroConverter" could not be found.
dev-kafka-connect        |      at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:715)
dev-kafka-connect        |      at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:460)
dev-kafka-connect        |      at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453)
dev-kafka-connect        |      at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
dev-kafka-connect        |      at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
dev-kafka-connect        |      at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:218)
dev-kafka-connect        |      at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:272)
dev-kafka-connect        |      at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:72)

Here is an extract of my docker-compose file: 

    kafka-connect:
        image: confluentinc/cp-kafka-connect:latest
        restart: always
        container_name: dev-kafka-connect
        depends_on:
            - zookeeper
            - kafka-broker
            - kafka-schema-registry
        env_file:
            - ${ENV_BASE_DIR}/confluent/global.env
            - ${ENV_BASE_DIR}/confluent/kafka-connect.env
        network_mode: host
        hostname: localhost
        extra_hosts:
            - "moby:127.0.0.1"
        volumes:
            - /tmp/quickstart/file:/tmp/quickstart
            - /tmp/quickstart/jars:/etc/kafka-connect/jars

Here is my variables (cat of kafka-connect.env): 

CONTROL_CENTER_ZOOKEEPER_CONNECT=127.0.0.1:2181
CONTROL_CENTER_BOOTSTRAP_SERVERS=127.0.0.1:9092
CONTROL_CENTER_REPLICATION_FACTOR=1
CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS=1
CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS=1
CONTROL_CENTER_STREAMS_NUM_STREAM_THREADS=1
CONTROL_CENTER_CONNECT_CLUSTER=http://127.0.0.1:8082
CONTROL_CENTER_LOG4J_LOGGERS="kafka.controller=ERROR"
CONTROL_CENTER_LOG4J_ROOT_LOGLEVEL=ERROR
CONTROL_CENTER_TOOLS_LOG4J_LOGLEVEL=ERROR

Bellow, some additional information's:

Server: Engine: Version: 18.01.0-ce API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: 03596f51b1 Built: Sun Jan 14 23:11:14 2018 OS/Arch: linux/amd64 Experimental: false

- `docker-compose --version`

docker-compose version 1.18.0, build unknown



Thanks for your help.
JR
OneCricketeer commented 6 years ago

I don't think your connect.env file is correct... The CONTROL_CENTER variables won't be loaded. Plus, I think you are missing CONNECT_PLUGIN_PATH, or have overwritten it or CLASSPATH

toniiiik commented 5 years ago

I have the same problem as @teaguecole described. When I run kafka connect from docker-compose I got the error:

org.apache.kafka.common.config.ConfigException: Invalid value "io.confluent.connect.avro.AvroConverter" for configuration key.converter: Class "io.confluent.connect.avro.AvroConverter" could not be found.
connect_1  |    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:724)
connect_1  |    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:469)
connect_1  |    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:462)
connect_1  |    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
connect_1  |    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
connect_1  |    at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:345)
connect_1  |    at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:272)
connect_1  |    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:76)
demo_connect_1 exited with code 2

here is my docker-compose:

version: '3.1'
services:
  zoo:
    image: confluentinc/cp-zookeeper:5.0.1
    environment:
    - ZOOKEEPER_CLIENT_PORT=2181 
    - ZOOKEEPER_TICK_TIME=2000
    ports:
    - 2181:2181
  kafka:
    image: confluentinc/cp-kafka:5.0.1
    ports:
    - 9092:9092
    environment:
    - KAFKA_ZOOKEEPER_CONNECT=zoo:2181 
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 
    - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 
    links:
    - zoo
  schema:
    image: confluentinc/cp-schema-registry:5.0.1
    environment: 
    - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zoo:2181
    - SCHEMA_REGISTRY_HOST_NAME=schema
    - SCHEMA_REGISTRY_LISTENERS=http://schema:8081
    links:
    - zoo
    ports:
    - 8081:8081
  connect:
    image: confluentinc/cp-kafka-connect:latest
    environment:
    - CONNECT_BOOTSTRAP_SERVERS=kafka:9092
    - CONNECT_REST_PORT=8083
    - CONNECT_GROUP_ID="quickstart-avro"
    - CONNECT_CONFIG_STORAGE_TOPIC="quickstart-avro-config"
    - CONNECT_OFFSET_STORAGE_TOPIC="quickstart-avro-offsets"
    - CONNECT_STATUS_STORAGE_TOPIC="quickstart-avro-status"
    - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
    - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
    - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
    - CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter"
    - CONNECT_VALUE_CONVERTER="io.confluent.connect.avro.AvroConverter"
    - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL="http://schema:8081"
    - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL="http://schema:8081"
    - CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter"
    - CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter"
    - CONNECT_REST_ADVERTISED_HOST_NAME="localhost"
    - CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG
    - CONNECT_PLUGIN_PATH=/usr/share/java/
    links:
    - schema
    - kafka
    ports:
    - 8083:8083

when I run connect outside of compose with the following command:

docker run -d \
  --name=kafka-connect-avro \
  --net=demo_default \
  -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
  -e CONNECT_REST_PORT=28083 \
  -e CONNECT_GROUP_ID="quickstart-avro" \
  -e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-avro-config" \
  -e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-avro-offsets" \
  -e CONNECT_STATUS_STORAGE_TOPIC="quickstart-avro-status" \
  -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter" \
  -e CONNECT_VALUE_CONVERTER="io.confluent.connect.avro.AvroConverter" \
  -e CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL="http://zoo:8081" \
  -e CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL="http://zoo:8081" \
  -e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_REST_ADVERTISED_HOST_NAME="localhost" \
  -e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
  -e CONNECT_PLUGIN_PATH=/usr/share/java \
  confluentinc/cp-kafka-connect:latest

...it works.

edwardpang-computime commented 5 years ago

same to @toniiiik & @teaguecole

docker-compose fail, but same config works using command line docker run

error says:

[2019-01-06 08:38:31,246] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
org.apache.kafka.common.config.ConfigException: Invalid value "org.apache.kafka.connect.json.JsonConverter" for configuration key.converter: Class "org.apache.kafka.connect.json.JsonConverter" could not be found.
        at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:718)
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:471)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:464)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
        at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:360)
        at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:272)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:76)
ybyzek commented 5 years ago

@toniiiik ,@edwardpang-computime : try removing the double quotes in the value.

For example, instead of 

    - CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter"

use

    - CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter

etc ...

edwardpang-computime commented 5 years ago

@ybyzek THX, it works