search

confluentinc / cp-all-in-one

docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform
965 stars 684 forks source link

Confluent Platform Quick Start (Docker): Datagen Connector not found #10

Open derekgreer opened 4 years ago

derekgreer commented 4 years ago

Running Docker for windows.

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Windows Version 10.0.18362 Build 18362

I'm working through the Confluent Platform Quick Start (Docker).

Initially, step 2 shows all containers running, but eventually the connect container exits:

$ docker-compose ps
     Name                    Command                  State                         Ports
------------------------------------------------------------------------------------------------------------
broker            /etc/confluent/docker/run        Up             0.0.0.0:9092->9092/tcp
connect           /etc/confluent/docker/run        Exit 137
control-center    /etc/confluent/docker/run        Up             0.0.0.0:9021->9021/tcp
ksql-datagen      bash -c echo Waiting for K ...   Up
ksqldb-cli        /bin/sh                          Up
ksqldb-server     /etc/confluent/docker/run        Up (healthy)   0.0.0.0:8088->8088/tcp
rest-proxy        /etc/confluent/docker/run        Up             0.0.0.0:8082->8082/tcp
schema-registry   /etc/confluent/docker/run        Up             0.0.0.0:8081->8081/tcp
zookeeper         /etc/confluent/docker/run        Up             0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

For step 3, the Datagen Connector isn't found. I consulted the referenced Issue: Cannot locate the Datagen Connector, but the resolutions do not produce the desired results.

Upon issuing the following command: docker-compose build --no-cache connect

The output is as follows:

connect uses an image, skipping

Upon issuing the following command: docker-compose logs connect | grep -i Datagen

There is no output.

Upon issuing the following command: docker-compose exec connect ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen/lib/

The output is as follows:

ERROR: No container found for connect_1

If I attempt to restart the connect container and issue the above command again before it exits, it does produce the following:

$ docker-compose exec connect ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen/lib/
automaton-1.11-8.jar             generex-1.0.2.jar                     jakarta.annotation-api-1.3.5.jar  jersey-client-2.30.jar             kafka-schema-registry-client-5.5.0.jar
avro-1.9.2.jar                   guava-18.0.jar                        jakarta.el-3.0.2.jar              jersey-common-2.30.jar             kafka-schema-serializer-5.5.0.jar
avro-random-generator-0.3.1.jar  hibernate-validator-6.0.17.Final.jar  jakarta.el-api-3.0.3.jar          jersey-media-jaxb-2.30.jar         osgi-resource-locator-1.0.3.jar
classmate-1.3.4.jar              jackson-annotations-2.10.2.jar        jakarta.inject-2.6.1.jar          jersey-server-2.30.jar             slf4j-api-1.7.26.jar
common-config-5.5.0.jar          jackson-core-2.10.2.jar               jakarta.validation-api-2.0.2.jar  joda-time-2.9.9.jar                snakeyaml-1.24.jar
common-utils-5.5.0.jar           jackson-databind-2.10.2.jar           jakarta.ws.rs-api-2.1.6.jar       kafka-avro-serializer-5.5.0.jar    swagger-annotations-1.5.22.jar
commons-compress-1.19.jar        jackson-dataformat-yaml-2.10.2.jar    jboss-logging-3.3.2.Final.jar     kafka-connect-avro-data-5.5.0.jar  swagger-core-1.5.3.jar
commons-lang3-3.2.1.jar          jackson-datatype-joda-2.10.2.jar      jersey-bean-validation-2.30.jar   kafka-connect-datagen-0.3.2.jar    swagger-models-1.5.3.jar
ybyzek commented 4 years ago

@derekgreer

  1. cp-all-in-one has been verified on macOS and other operating systems
  2. Confluent Platform is not supported on Windows (even with Docker): https://docs.confluent.io/current/installation/versions-interoperability.html#docker so there may be unexpected behavior. Please switch to a supported OS.
  3. If you want to still try to troubleshoot in the unsupported Windows OS, check the full output of docker-compose logs connect?
RenlySir commented 4 years ago

i use centos 7.5 also get this issue, so what should i do, the machine memory is 4g,not enough?

ybyzek commented 4 years ago

@RenlySir what error messages are you seeing in your environment? Please use docker-compose logs to troubleshoot

derekgreer commented 4 years ago

Strange ... I gave up yesterday and decided to just play around with the kafka distro directly. I didn't have Java installed, so I had to install that. This morning, I tried running this again to look at the logs and lo and behold it's now working. Is it possible there is some dependency of the containers for Java to be installed on the host operating system?

azimshaik commented 4 years ago

@ybyzek is it still the case Windows is not supported?

abbringas commented 4 years ago

I am learning kafka by following this artitle Quick Start for Apache Kafka using Confluent Platform (Docker) https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html#ce-docker-quickstart

But I think i also have some issues using the datagenConnector and I dont know how I can fix it. See information below.

Admin@DCNI-014CS5J-LX MINGW64 /c/Code/Tools/cp-all-in-one/cp-all-in-one (5.5.1-post) $ docker-compose build --no-cache connect connect uses an image, skipping

Admin@DCNI-014CS5J-LX MINGW64 /c/Code/Tools/cp-all-in-one/cp-all-in-one (5.5.1-post) $ docker-compose exec connect ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen ls: cannot access C:/Program Files/Git/usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen: No such file or directory

Note: I am also using windows 10.

OneCricketeer commented 3 years ago

@azimshaik Correct, Windows is not; the Linux containers are

@abbringas You are missing quotes, otherwise, it assumes a local file path...

docker-compose exec connect "ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen"
OneCricketeer commented 3 years ago

@derekgreer

Is it possible there is some dependency of the containers for Java to be installed on the host operating system?

Unlikely. Docker itself does not require Java, so therefore the containers have no external reliance on it either.

Connect container will exit because you need to configure Docker with more memory (at least 6GB, as mentioned in the linked docs)

Sounds like you've fixed your issue, though, so this can be closed?