search

udacity / nd029-c2-apache-spark-and-spark-streaming-starter

This is the starter code for both the course and the project for Data Streaming with Spark
Other
14 stars 38 forks source link

`gcr.io/simulation-images/kafka-connect-redis-source` container error #1

Open rightx2 opened 2 years ago

rightx2 commented 2 years ago

ENV

Problem I've cloned the repo and run docker-compose up, all containers are healthy except the connect service, which uses gcr.io/simulation-images/kafka-connect-redis-source image:

CONTAINER ID   IMAGE                                                 COMMAND                  CREATED         STATUS                            PORTS                                                                                            NAMES
fbd12e6e5264   gcr.io/simulation-images/kafka-connect-redis-source   "/bin/sh -c 'connect…"   3 minutes ago   Up 3 minutes (health: starting)   0.0.0.0:5005->5005/tcp, :::5005->5005/tcp, 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp   nd029-c2-apache-spark-and-spark-streaming-starter_connect_1
f4b45d4f2930   gcr.io/simulation-images/trucking-simulation          "java -jar /trucking…"   3 minutes ago   Up 3 minutes                      4567/tcp                                                                                         nd029-c2-apache-spark-and-spark-streaming-starter_trucking-simulation_1
1ccf3fc356d1   gcr.io/simulation-images/banking-simulation           "java -jar /banking.…"   3 minutes ago   Up 3 minutes                      4567/tcp                                                                                         nd029-c2-apache-spark-and-spark-streaming-starter_banking-simulation_1
c2a2ca10f99b   gcr.io/simulation-images/stedi                        "java -jar /stedi.jar"   3 minutes ago   Up 3 minutes                      0.0.0.0:4567->4567/tcp, :::4567->4567/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_stedi_1
f3f3d94276e3   confluentinc/cp-kafka:5.5.1                           "/etc/confluent/dock…"   3 minutes ago   Up 3 minutes                      0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_kafka_1
27a7748a170a   confluentinc/cp-zookeeper:5.5.1                       "/etc/confluent/dock…"   3 minutes ago   Up 3 minutes                      2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp                                    nd029-c2-apache-spark-and-spark-streaming-starter_zookeeper_1
62f27d778172   redis:6.0.6                                           "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes                      0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_redis_1
c57142f3f12f   bitnami/spark:3-debian-10                             "/opt/bitnami/script…"   3 minutes ago   Up 3 minutes                      0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_spark_1

It kept restarting, so I checked the log of that container:

$ docker-compose logs connect
Attaching to nd029-c2-apache-spark-and-spark-streaming-starter_connect_1
connect_1              | [2021-11-30 23:34:27,045] INFO Kafka Connect standalone worker initializing ... (org.apache.kafka.connect.cli.ConnectStandalone:69)
connect_1              | [2021-11-30 23:34:27,054] INFO WorkerInfo values:
connect_1              |    jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/log/kafka, -Dlog4j.configuration=file:/etc/kafka/connect-log4j.properties
connect_1              |    jvm.spec = Azul Systems, Inc., OpenJDK 64-Bit Server VM, 1.8.0_212, 25.212-b04
connect_1              |    jvm.classpath = /usr/share/java/kafka-connect-redis-source/commons-logging-1.2.jar:/usr/share/java/kafka-connect-redis-source/jedis-2.9.0.jar:/usr/share/java/kafka-connect-redis-source/kafka-connect-redis-1.0-SNAPSHOT.jar:/usr/share/java/kafka-connect-redis-source/redis-replicator-3.0.1.jar:/usr/share/java/kafka/activation-1.1.1.jar:/usr/share/java/kafka/aopalliance-repackaged-2.5.0.jar:/usr/share/java/kafka/argparse4j-0.7.0.jar:/usr/share/java/kafka/audience-annotations-0.5.0.jar:/usr/share/java/kafka/avro-1.9.2.jar:/usr/share/java/kafka/commons-cli-1.4.jar:/usr/share/java/kafka/commons-codec-1.11.jar:/usr/share/java/kafka/commons-compress-1.19.jar:/usr/share/java/kafka/commons-lang3-3.8.1.jar:/usr/share/java/kafka/commons-logging-1.2.jar:/usr/share/java/kafka/connect-api-5.5.1-ccs.jar:/usr/share/java/kafka/connect-basic-auth-extension-5.5.1-ccs.jar:/usr/share/java/kafka/connect-file-5.5.1-ccs.jar:/usr/share/java/kafka/connect-json-5.5.1-ccs.jar:/usr/share/java/kafka/connect-mirror-5.5.1-ccs.jar:/usr/share/java/kafka/connect-mirror-client-5.5.1-ccs.jar:/usr/share/java/kafka/connect-runtime-5.5.1-ccs.jar:/usr/share/java/kafka/connect-transforms-5.5.1-ccs.jar:/usr/share/java/kafka/hk2-api-2.5.0.jar:/usr/share/java/kafka/hk2-locator-2.5.0.jar:/usr/share/java/kafka/hk2-utils-2.5.0.jar:/usr/share/java/kafka/httpclient-4.5.11.jar:/usr/share/java/kafka/httpcore-4.4.13.jar:/usr/share/java/kafka/httpmime-4.5.11.jar:/usr/share/java/kafka/jackson-annotations-2.10.2.jar:/usr/share/java/kafka/jackson-core-2.10.2.jar:/usr/share/java/kafka/jackson-databind-2.10.2.jar:/usr/share/java/kafka/jackson-dataformat-csv-2.10.2.jar:/usr/share/java/kafka/jackson-datatype-jdk8-2.10.2.jar:/usr/share/java/kafka/jackson-jaxrs-base-2.10.2.jar:/usr/share/java/kafka/jackson-jaxrs-json-provider-2.10.2.jar:/usr/share/java/kafka/jackson-module-jaxb-annotations-2.10.2.jar:/usr/share/java/kafka/jackson-module-paranamer-2.10.2.jar:/usr/share/java/kafka/jackson-module-scala_2.12-2.10.2.jar:/usr/share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/share/java/kafka/jakarta.annotation-api-1.3.4.jar:/usr/share/java/kafka/jakarta.inject-2.5.0.jar:/usr/share/java/kafka/jakarta.ws.rs-api-2.1.5.jar:/usr/share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/share/java/kafka/javassist-3.22.0-CR2.jar:/usr/share/java/kafka/javassist-3.26.0-GA.jar:/usr/share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/share/java/kafka/jaxb-api-2.3.0.jar:/usr/share/java/kafka/jersey-client-2.28.jar:/usr/share/java/kafka/jersey-common-2.28.jar:/usr/share/java/kafka/jersey-container-servlet-2.28.jar:/usr/share/java/kafka/jersey-container-servlet-core-2.28.jar:/usr/share/java/kafka/jersey-hk2-2.28.jar:/usr/share/java/kafka/jersey-media-jaxb-2.28.jar:/usr/share/java/kafka/jersey-server-2.28.jar:/usr/share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/share/java/kafka/jopt-simple-5.0.4.jar:/usr/share/java/kafka/kafka-clients-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-log4j-appender-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-examples-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-scala_2.12-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-test-utils-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-tools-5.5.1-ccs.jar:/usr/share/java/kafka/kafka.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-javadoc.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-scaladoc.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-sources.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-test-sources.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-test.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs.jar:/usr/share/java/kafka/log4j-1.2.17.jar:/usr/share/java/kafka/lz4-java-1.7.1.jar:/usr/share/java/kafka/maven-artifact-3.6.3.jar:/usr/share/java/kafka/metrics-core-2.2.0.jar:/usr/share/java/kafka/netty-buffer-4.1.48.Final.jar:/usr/share/java/kafka/netty-codec-4.1.48.Final.jar:/usr/share/java/kafka/netty-common-4.1.48.Final.jar:/usr/share/java/kafka/netty-handler-4.1.48.Final.jar:/usr/share/java/kafka/netty-resolver-4.1.48.Final.jar:/usr/share/java/kafka/netty-transport-4.1.48.Final.jar:/usr/share/java/kafka/netty-transport-native-epoll-4.1.48.Final.jar:/usr/share/java/kafka/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/share/java/kafka/paranamer-2.8.jar:/usr/share/java/kafka/plexus-utils-3.2.1.jar:/usr/share/java/kafka/reflections-0.9.12.jar:/usr/share/java/kafka/rocksdbjni-5.18.3.jar:/usr/share/java/kafka/scala-collection-compat_2.12-2.1.3.jar:/usr/share/java/kafka/scala-java8-compat_2.12-0.9.0.jar:/usr/share/java/kafka/scala-library-2.12.10.jar:/usr/share/java/kafka/scala-logging_2.12-3.9.2.jar:/usr/share/java/kafka/scala-reflect-2.12.10.jar:/usr/share/java/kafka/slf4j-api-1.7.30.jar:/usr/share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/share/java/kafka/snappy-java-1.1.7.3.jar:/usr/share/java/kafka/support-metrics-client-5.5.1-ccs.jar:/usr/share/java/kafka/support-metrics-common-5.5.1-ccs.jar:/usr/share/java/kafka/validation-api-2.0.1.Final.jar:/usr/share/java/kafka/zookeeper-3.5.8.jar:/usr/share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/share/java/kafka/confluent-metrics-5.5.1-ce.jar:/usr/share/java/confluent-common/build-tools-5.5.1.jar:/usr/share/java/confluent-common/common-config-5.5.1.jar:/usr/share/java/confluent-common/common-metrics-5.5.1.jar:/usr/share/java/confluent-common/common-utils-5.5.1.jar:/usr/share/java/confluent-common/slf4j-api-1.7.26.jar:/usr/share/java/kafka-serde-tools/animal-sniffer-annotations-1.14.jar:/usr/share/java/kafka-serde-tools/annotations-13.0.jar:/usr/share/java/kafka-serde-tools/avro-1.9.2.jar:/usr/share/java/kafka-serde-tools/checker-compat-qual-2.0.0.jar:/usr/share/java/kafka-serde-tools/classgraph-4.8.21.jar:/usr/share/java/kafka-serde-tools/commons-collections-3.2.2.jar:/usr/share/java/kafka-serde-tools/commons-compress-1.19.jar:/usr/share/java/kafka-serde-tools/commons-digester-1.8.1.jar:/usr/share/java/kafka-serde-tools/commons-logging-1.2.jar:/usr/share/java/kafka-serde-tools/commons-validator-1.6.jar:/usr/share/java/kafka-serde-tools/error_prone_annotations-2.3.4.jar:/usr/share/java/kafka-serde-tools/gson-2.8.5.jar:/usr/share/java/kafka-serde-tools/guava-24.0-jre.jar:/usr/share/java/kafka-serde-tools/handy-uri-templates-2.1.8.jar:/usr/share/java/kafka-serde-tools/j2objc-annotations-1.1.jar:/usr/share/java/kafka-serde-tools/jackson-annotations-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-core-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-databind-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-guava-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-jdk8-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-joda-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-jsr310-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-module-parameter-names-2.10.2.jar:/usr/share/java/kafka-serde-tools/jakarta.annotation-api-1.3.5.jar:/usr/share/java/kafka-serde-tools/jakarta.inject-2.6.1.jar:/usr/share/java/kafka-serde-tools/jakarta.ws.rs-api-2.1.6.jar:/usr/share/java/kafka-serde-tools/javax.ws.rs-api-2.1.1.jar:/usr/share/java/kafka-serde-tools/jersey-common-2.30.jar:/usr/share/java/kafka-serde-tools/joda-time-2.9.9.jar:/usr/share/java/kafka-serde-tools/json-20190722.jar:/usr/share/java/kafka-serde-tools/jsr305-1.3.9.jar:/usr/share/java/kafka-serde-tools/kafka-avro-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-avro-converter-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-avro-data-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-json-schema-converter-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-protobuf-converter-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-json-schema-provider-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-json-schema-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-json-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-protobuf-provider-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-protobuf-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-schema-registry-client-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-schema-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-streams-5.5.1-ccs.jar:/usr/share/java/kafka-serde-tools/kafka-streams-avro-serde-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-streams-json-schema-serde-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-streams-protobuf-serde-5.5.1.jar:/usr/share/java/kafka-serde-tools/kotlin-reflect-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-script-runtime-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-common-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-compiler-embeddable-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-compiler-impl-embeddable-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-jvm-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-common-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-jdk7-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-jdk8-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlinx-coroutines-core-1.1.1.jar:/usr/share/java/kafka-serde-tools/kotlinx-coroutines-core-common-1.1.1.jar:/usr/share/java/kafka-serde-tools/mbknor-jackson-jsonschema_2.12-1.0.39.jar:/usr/share/java/kafka-serde-tools/okio-2.5.0.jar:/usr/share/java/kafka-serde-tools/org.everit.json.schema-1.12.1.jar:/usr/share/java/kafka-serde-tools/osgi-resource-locator-1.0.3.jar:/usr/share/java/kafka-serde-tools/protobuf-java-3.11.4.jar:/usr/share/java/kafka-serde-tools/protobuf-java-util-3.11.4.jar:/usr/share/java/kafka-serde-tools/re2j-1.3.jar:/usr/share/java/kafka-serde-tools/rocksdbjni-5.18.3.jar:/usr/share/java/kafka-serde-tools/scala-library-2.12.10.jar:/usr/share/java/kafka-serde-tools/swagger-annotations-1.6.0.jar:/usr/share/java/kafka-serde-tools/validation-api-2.0.1.Final.jar:/usr/share/java/kafka-serde-tools/wire-runtime-3.2.2.jar:/usr/share/java/kafka-serde-tools/wire-schema-3.2.2.jar:/usr/share/java/monitoring-interceptors/monitoring-interceptors-5.5.1.jar:/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.5.0.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/avro-1.9.2.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-codec-1.11.jar:/usr/bin/../share/java/kafka/commons-compress-1.19.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/commons-logging-1.2.jar:/usr/bin/../share/java/kafka/connect-api-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-file-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-json-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.5.0.jar:/usr/bin/../share/java/kafka/hk2-locator-2.5.0.jar:/usr/bin/../share/java/kafka/hk2-utils-2.5.0.jar:/usr/bin/../share/java/kafka/httpclient-4.5.11.jar:/usr/bin/../share/java/kafka/httpcore-4.4.13.jar:/usr/bin/../share/java/kafka/httpmime-4.5.11.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-core-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-databind-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-module-paranamer-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.12-2.10.2.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.4.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.5.0.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.5.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/bin/../share/java/kafka/javassist-3.22.0-CR2.jar:/usr/bin/../share/java/kafka/javassist-3.26.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.28.jar:/usr/bin/../share/java/kafka/jersey-common-2.28.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.28.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.28.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.28.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.28.jar:/usr/bin/../share/java/kafka/jersey-server-2.28.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.12-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-javadoc.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-scaladoc.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-sources.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-test-sources.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-test.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/log4j-1.2.17.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.6.3.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.18.3.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.12-2.1.3.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.12-0.9.0.jar:/usr/bin/../share/java/kafka/scala-library-2.12.10.jar:/usr/bin/../share/java/kafka/scala-logging_2.12-3.9.2.jar:/usr/bin/../share/java/kafka/scala-reflect-2.12.10.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.30.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.7.3.jar:/usr/bin/../share/java/kafka/support-metrics-client-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/support-metrics-common-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/validation-api-2.0.1.Final.jar:/usr/bin/../share/java/kafka/zookeeper-3.5.8.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/bin/../share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/bin/../share/java/kafka/confluent-metrics-5.5.1-ce.jar:/usr/bin/../support-metrics-client/build/dependant-libs-2.12/*:/usr/bin/../support-metrics-client/build/libs/*:/usr/share/java/support-metrics-client/*
connect_1              |    os.spec = Linux, amd64, 5.11.0-38-generic
connect_1              |    os.vcpus = 16
connect_1              |  (org.apache.kafka.connect.runtime.WorkerInfo:71)
connect_1              | [2021-11-30 23:34:27,062] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectStandalone:78)
connect_1              | [2021-11-30 23:34:29,761] INFO Registered loader: sun.misc.Launcher$AppClassLoader@764c12b6 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:262)
connect_1              | [2021-11-30 23:34:29,761] INFO Added plugin 'org.apache.kafka.connect.redis.RedisSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,762] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,762] INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,762] INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,769] INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,772] INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,772] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,772] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,774] INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,774] INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'io.confluent.connect.json.JsonSchemaConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'io.confluent.connect.protobuf.ProtobufConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)
connect_1              | [2021-11-30 23:34:29,782] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,782] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,783] INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,783] INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,783] INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,783] INFO Added aliases 'RedisSourceConnector' and 'RedisSource' to plugin 'org.apache.kafka.connect.redis.RedisSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,783] INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,784] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,784] INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,784] INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,784] INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,784] INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,784] INFO Added aliases 'AvroConverter' and 'Avro' to plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'JsonSchemaConverter' and 'JsonSchema' to plugin 'io.confluent.connect.json.JsonSchemaConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'ProtobufConverter' and 'Protobuf' to plugin 'io.confluent.connect.protobuf.ProtobufConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,785] INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,786] INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,787] INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,787] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,787] INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)
connect_1              | [2021-11-30 23:34:29,787] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,788] INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)
connect_1              | [2021-11-30 23:34:29,788] INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)
connect_1              | [2021-11-30 23:34:29,788] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)
connect_1              | [2021-11-30 23:34:29,788] INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)
connect_1              | [2021-11-30 23:34:29,788] INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,789] INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,789] INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)
connect_1              | [2021-11-30 23:34:29,806] INFO StandaloneConfig values:
connect_1              |    access.control.allow.methods =
connect_1              |    access.control.allow.origin =
connect_1              |    admin.listeners = null
connect_1              |    bootstrap.servers = [kafka:19092]
connect_1              |    client.dns.lookup = default
connect_1              |    config.providers = []
connect_1              |    connector.client.config.override.policy = None
connect_1              |    header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
connect_1              |    internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
connect_1              |    internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
connect_1              |    key.converter = class org.apache.kafka.connect.converters.ByteArrayConverter
connect_1              |    listeners = null
connect_1              |    metric.reporters = []
connect_1              |    metrics.num.samples = 2
connect_1              |    metrics.recording.level = INFO
connect_1              |    metrics.sample.window.ms = 30000
connect_1              |    offset.flush.interval.ms = 60000
connect_1              |    offset.flush.timeout.ms = 5000
connect_1              |    offset.storage.file.filename = /tmp/connect.offsets
connect_1              |    plugin.path = [share/java, /opt/confluent-5.1.0/share/confluent-hub-components]
connect_1              |    rest.advertised.host.name = null
connect_1              |    rest.advertised.listener = null
connect_1              |    rest.advertised.port = null
connect_1              |    rest.extension.classes = []
connect_1              |    rest.host.name = null
connect_1              |    rest.port = 8083
connect_1              |    ssl.cipher.suites = null
connect_1              |    ssl.client.auth = none
connect_1              |    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
connect_1              |    ssl.endpoint.identification.algorithm = https
connect_1              |    ssl.key.password = null
connect_1              |    ssl.keymanager.algorithm = SunX509
connect_1              |    ssl.keystore.location = null
connect_1              |    ssl.keystore.password = null
connect_1              |    ssl.keystore.type = JKS
connect_1              |    ssl.protocol = TLS
connect_1              |    ssl.provider = null
connect_1              |    ssl.secure.random.implementation = null
connect_1              |    ssl.trustmanager.algorithm = PKIX
connect_1              |    ssl.truststore.location = null
connect_1              |    ssl.truststore.password = null
connect_1              |    ssl.truststore.type = JKS
connect_1              |    task.shutdown.graceful.timeout.ms = 5000
connect_1              |    topic.tracking.allow.reset = true
connect_1              |    topic.tracking.enable = true
connect_1              |    value.converter = class org.apache.kafka.connect.storage.StringConverter
connect_1              |  (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:347)
connect_1              | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)
connect_1              | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.key.converter.schemas.enable' (along with all configuration for 'internal.key.converter') is deprecated and may be removed in an upcoming release. The specified value 'false' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)
connect_1              | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)
connect_1              | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.value.converter.schemas.enable' (along with all configuration for 'internal.value.converter') is deprecated and may be removed in an upcoming release. The specified value 'false' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)
connect_1              | [2021-11-30 23:34:29,808] INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils:43)
connect_1              | [2021-11-30 23:34:29,811] INFO AdminClientConfig values:
connect_1              |    bootstrap.servers = [kafka:19092]
connect_1              |    client.dns.lookup = default
connect_1              |    client.id =
connect_1              |    connections.max.idle.ms = 300000
connect_1              |    default.api.timeout.ms = 60000
connect_1              |    metadata.max.age.ms = 300000
connect_1              |    metric.reporters = []
connect_1              |    metrics.num.samples = 2
connect_1              |    metrics.recording.level = INFO
connect_1              |    metrics.sample.window.ms = 30000
connect_1              |    receive.buffer.bytes = 65536
connect_1              |    reconnect.backoff.max.ms = 1000
connect_1              |    reconnect.backoff.ms = 50
connect_1              |    request.timeout.ms = 30000
connect_1              |    retries = 2147483647
connect_1              |    retry.backoff.ms = 100
connect_1              |    sasl.client.callback.handler.class = null
connect_1              |    sasl.jaas.config = null
connect_1              |    sasl.kerberos.kinit.cmd = /usr/bin/kinit
connect_1              |    sasl.kerberos.min.time.before.relogin = 60000
connect_1              |    sasl.kerberos.service.name = null
connect_1              |    sasl.kerberos.ticket.renew.jitter = 0.05
connect_1              |    sasl.kerberos.ticket.renew.window.factor = 0.8
connect_1              |    sasl.login.callback.handler.class = null
connect_1              |    sasl.login.class = null
connect_1              |    sasl.login.refresh.buffer.seconds = 300
connect_1              |    sasl.login.refresh.min.period.seconds = 60
connect_1              |    sasl.login.refresh.window.factor = 0.8
connect_1              |    sasl.login.refresh.window.jitter = 0.05
connect_1              |    sasl.mechanism = GSSAPI
connect_1              |    security.protocol = PLAINTEXT
connect_1              |    security.providers = null
connect_1              |    send.buffer.bytes = 131072
connect_1              |    ssl.cipher.suites = null
connect_1              |    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
connect_1              |    ssl.endpoint.identification.algorithm = https
connect_1              |    ssl.key.password = null
connect_1              |    ssl.keymanager.algorithm = SunX509
connect_1              |    ssl.keystore.location = null
connect_1              |    ssl.keystore.password = null
connect_1              |    ssl.keystore.type = JKS
connect_1              |    ssl.protocol = TLS
connect_1              |    ssl.provider = null
connect_1              |    ssl.secure.random.implementation = null
connect_1              |    ssl.trustmanager.algorithm = PKIX
connect_1              |    ssl.truststore.location = null
connect_1              |    ssl.truststore.password = null
connect_1              |    ssl.truststore.type = JKS
connect_1              |  (org.apache.kafka.clients.admin.AdminClientConfig:347)
connect_1              | [2021-11-30 23:34:29,860] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,860] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,860] WARN The configuration 'max.request.size' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,860] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,860] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,860] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,861] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,861] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,861] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,861] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,861] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
connect_1              | [2021-11-30 23:34:29,861] INFO Kafka version: 5.5.1-ccs (org.apache.kafka.common.utils.AppInfoParser:117)
connect_1              | [2021-11-30 23:34:29,861] INFO Kafka commitId: 3c4783aac9e33249 (org.apache.kafka.common.utils.AppInfoParser:118)
connect_1              | [2021-11-30 23:34:29,861] INFO Kafka startTimeMs: 1638315269861 (org.apache.kafka.common.utils.AppInfoParser:119)
connect_1              | [2021-11-30 23:34:29,879] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)
connect_1              | [2021-11-30 23:34:29,981] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)
connect_1              | [2021-11-30 23:34:30,083] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)
connect_1              | [2021-11-30 23:34:30,385] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)
connect_1              | [2021-11-30 23:34:31,212] INFO Kafka cluster ID: 7Rv27TYRSA2ZLyiWx0JFQA (org.apache.kafka.connect.util.ConnectUtils:59)
connect_1              | [2021-11-30 23:34:31,236] INFO Logging initialized @4693ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:169)
connect_1              | [2021-11-30 23:34:31,300] INFO Added connector for http://:8083 (org.apache.kafka.connect.runtime.rest.RestServer:131)
connect_1              | [2021-11-30 23:34:31,301] INFO Initializing REST server (org.apache.kafka.connect.runtime.rest.RestServer:203)
connect_1              | [2021-11-30 23:34:31,310] INFO jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 1.8.0_212-b04 (org.eclipse.jetty.server.Server:359)
connect_1              | [2021-11-30 23:34:31,344] INFO Started http_8083@d176a31{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:330)
connect_1              | [2021-11-30 23:34:31,344] INFO Started @4802ms (org.eclipse.jetty.server.Server:399)
connect_1              | [2021-11-30 23:34:31,370] INFO Advertised URI: http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:365)
connect_1              | [2021-11-30 23:34:31,371] INFO REST server listening at http://172.29.0.10:8083/, advertising URL http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:218)
connect_1              | [2021-11-30 23:34:31,372] INFO Advertised URI: http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:365)
connect_1              | [2021-11-30 23:34:31,374] INFO REST admin endpoints at http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:219)
connect_1              | [2021-11-30 23:34:31,375] INFO Advertised URI: http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:365)
connect_1              | [2021-11-30 23:34:31,375] INFO Setting up None Policy for ConnectorClientConfigOverride. This will disallow any client configuration to be overridden (org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy:45)
connect_1              | [2021-11-30 23:34:31,393] INFO Kafka version: 5.5.1-ccs (org.apache.kafka.common.utils.AppInfoParser:117)
connect_1              | [2021-11-30 23:34:31,393] INFO Kafka commitId: 3c4783aac9e33249 (org.apache.kafka.common.utils.AppInfoParser:118)
connect_1              | [2021-11-30 23:34:31,394] INFO Kafka startTimeMs: 1638315271393 (org.apache.kafka.common.utils.AppInfoParser:119)
connect_1              | [2021-11-30 23:34:31,581] INFO JsonConverterConfig values:
connect_1              |    converter.type = key
connect_1              |    decimal.format = BASE64
connect_1              |    schemas.cache.size = 1000
connect_1              |    schemas.enable = false
connect_1              |  (org.apache.kafka.connect.json.JsonConverterConfig:347)
connect_1              | [2021-11-30 23:34:31,583] INFO JsonConverterConfig values:
connect_1              |    converter.type = value
connect_1              |    decimal.format = BASE64
connect_1              |    schemas.cache.size = 1000
connect_1              |    schemas.enable = false
connect_1              |  (org.apache.kafka.connect.json.JsonConverterConfig:347)
connect_1              | [2021-11-30 23:34:31,597] INFO Kafka Connect standalone worker initialization took 4550ms (org.apache.kafka.connect.cli.ConnectStandalone:100)
connect_1              | [2021-11-30 23:34:31,598] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:51)
connect_1              | [2021-11-30 23:34:31,599] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:93)
connect_1              | [2021-11-30 23:34:31,599] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:184)
connect_1              | [2021-11-30 23:34:31,600] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:58)
connect_1              | [2021-11-30 23:34:31,604] INFO Worker started (org.apache.kafka.connect.runtime.Worker:191)
connect_1              | [2021-11-30 23:34:31,604] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:95)
connect_1              | [2021-11-30 23:34:31,605] INFO Initializing REST resources (org.apache.kafka.connect.runtime.rest.RestServer:223)
connect_1              | [2021-11-30 23:34:31,668] INFO Adding admin resources to main listener (org.apache.kafka.connect.runtime.rest.RestServer:240)
connect_1              | [2021-11-30 23:34:31,743] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session:333)
connect_1              | [2021-11-30 23:34:31,743] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session:338)
connect_1              | [2021-11-30 23:34:31,744] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session:140)
connect_1              | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
connect_1              | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be ignored.
connect_1              | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
connect_1              | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.RootResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
connect_1              | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
connect_1              | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.
connect_1              | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
connect_1              | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored.
connect_1              | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.Errors logErrors
connect_1              | WARNING: The following warnings have been detected: WARNING: The (sub)resource method listLoggers in org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains empty path annotation.
connect_1              | WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
connect_1              | WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
connect_1              | WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
connect_1              | WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.
connect_1              |
connect_1              | [2021-11-30 23:34:32,290] INFO Started o.e.j.s.ServletContextHandler@46d567cb{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:825)
connect_1              | [2021-11-30 23:34:32,290] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:313)
connect_1              | [2021-11-30 23:34:32,290] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
connect_1              | [2021-11-30 23:34:32,296] INFO AbstractConfig values:
connect_1              |  (org.apache.kafka.common.config.AbstractConfig:347)
connect_1              | [2021-11-30 23:34:32,302] INFO ConnectorConfig values:
connect_1              |    config.action.reload = restart
connect_1              |    connector.class = org.apache.kafka.connect.redis.RedisSourceConnector
connect_1              |    errors.log.enable = false
connect_1              |    errors.log.include.messages = false
connect_1              |    errors.retry.delay.max.ms = 60000
connect_1              |    errors.retry.timeout = 0
connect_1              |    errors.tolerance = none
connect_1              |    header.converter = null
connect_1              |    key.converter = null
connect_1              |    name = redis-config
connect_1              |    tasks.max = 1
connect_1              |    transforms = []
connect_1              |    value.converter = null
connect_1              |  (org.apache.kafka.connect.runtime.ConnectorConfig:347)
connect_1              | [2021-11-30 23:34:32,303] INFO EnrichedConnectorConfig values:
connect_1              |    config.action.reload = restart
connect_1              |    connector.class = org.apache.kafka.connect.redis.RedisSourceConnector
connect_1              |    errors.log.enable = false
connect_1              |    errors.log.include.messages = false
connect_1              |    errors.retry.delay.max.ms = 60000
connect_1              |    errors.retry.timeout = 0
connect_1              |    errors.tolerance = none
connect_1              |    header.converter = null
connect_1              |    key.converter = null
connect_1              |    name = redis-config
connect_1              |    tasks.max = 1
connect_1              |    transforms = []
connect_1              |    value.converter = null
connect_1              |  (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
connect_1              | [2021-11-30 23:34:32,303] INFO Creating connector redis-config of type org.apache.kafka.connect.redis.RedisSourceConnector (org.apache.kafka.connect.runtime.Worker:253)
connect_1              | [2021-11-30 23:34:32,305] INFO Instantiated connector redis-config with version 5.5.1-ccs of type class org.apache.kafka.connect.redis.RedisSourceConnector (org.apache.kafka.connect.runtime.Worker:256)
connect_1              | [2021-11-30 23:34:32,306] INFO Finished creating connector redis-config (org.apache.kafka.connect.runtime.Worker:275)
connect_1              | [2021-11-30 23:34:32,307] INFO SourceConnectorConfig values:
connect_1              |    config.action.reload = restart
connect_1              |    connector.class = org.apache.kafka.connect.redis.RedisSourceConnector
connect_1              |    errors.log.enable = false
connect_1              |    errors.log.include.messages = false
connect_1              |    errors.retry.delay.max.ms = 60000
connect_1              |    errors.retry.timeout = 0
connect_1              |    errors.tolerance = none
connect_1              |    header.converter = null
connect_1              |    key.converter = null
connect_1              |    name = redis-config
connect_1              |    tasks.max = 1
connect_1              |    transforms = []
connect_1              |    value.converter = null
connect_1              |  (org.apache.kafka.connect.runtime.SourceConnectorConfig:347)
connect_1              | [2021-11-30 23:34:32,307] INFO EnrichedConnectorConfig values:
connect_1              |    config.action.reload = restart
connect_1              |    connector.class = org.apache.kafka.connect.redis.RedisSourceConnector
connect_1              |    errors.log.enable = false
connect_1              |    errors.log.include.messages = false
connect_1              |    errors.retry.delay.max.ms = 60000
connect_1              |    errors.retry.timeout = 0
connect_1              |    errors.tolerance = none
connect_1              |    header.converter = null
connect_1              |    key.converter = null
connect_1              |    name = redis-config
connect_1              |    tasks.max = 1
connect_1              |    transforms = []
connect_1              |    value.converter = null
connect_1              |  (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
connect_1              | [2021-11-30 23:34:32,308] INFO Creating task redis-config-0 (org.apache.kafka.connect.runtime.Worker:421)
connect_1              | [2021-11-30 23:34:32,310] INFO ConnectorConfig values:
connect_1              |    config.action.reload = restart
connect_1              |    connector.class = org.apache.kafka.connect.redis.RedisSourceConnector
connect_1              |    errors.log.enable = false
connect_1              |    errors.log.include.messages = false
connect_1              |    errors.retry.delay.max.ms = 60000
connect_1              |    errors.retry.timeout = 0
connect_1              |    errors.tolerance = none
connect_1              |    header.converter = null
connect_1              |    key.converter = null
connect_1              |    name = redis-config
connect_1              |    tasks.max = 1
connect_1              |    transforms = []
connect_1              |    value.converter = null
connect_1              |  (org.apache.kafka.connect.runtime.ConnectorConfig:347)
connect_1              | [2021-11-30 23:34:32,310] INFO EnrichedConnectorConfig values:
connect_1              |    config.action.reload = restart
connect_1              |    connector.class = org.apache.kafka.connect.redis.RedisSourceConnector
connect_1              |    errors.log.enable = false
connect_1              |    errors.log.include.messages = false
connect_1              |    errors.retry.delay.max.ms = 60000
connect_1              |    errors.retry.timeout = 0
connect_1              |    errors.tolerance = none
connect_1              |    header.converter = null
connect_1              |    key.converter = null
connect_1              |    name = redis-config
connect_1              |    tasks.max = 1
connect_1              |    transforms = []
connect_1              |    value.converter = null
connect_1              |  (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
connect_1              | [2021-11-30 23:34:32,310] INFO TaskConfig values:
connect_1              |    task.class = class org.apache.kafka.connect.redis.RedisSourceTask
connect_1              |  (org.apache.kafka.connect.runtime.TaskConfig:347)
connect_1              | [2021-11-30 23:34:32,311] INFO Instantiated task redis-config-0 with version 5.5.1-ccs of type org.apache.kafka.connect.redis.RedisSourceTask (org.apache.kafka.connect.runtime.Worker:436)
connect_1              | [2021-11-30 23:34:32,311] INFO Set up the key converter class org.apache.kafka.connect.converters.ByteArrayConverter for task redis-config-0 using the worker config (org.apache.kafka.connect.runtime.Worker:449)
connect_1              | [2021-11-30 23:34:32,312] INFO StringConverterConfig values:
connect_1              |    converter.encoding = UTF8
connect_1              |    converter.type = value
connect_1              |  (org.apache.kafka.connect.storage.StringConverterConfig:347)
connect_1              | [2021-11-30 23:34:32,312] INFO Set up the value converter class org.apache.kafka.connect.storage.StringConverter for task redis-config-0 using the worker config (org.apache.kafka.connect.runtime.Worker:455)
connect_1              | [2021-11-30 23:34:32,312] INFO Set up the header converter class org.apache.kafka.connect.storage.SimpleHeaderConverter for task redis-config-0 using the worker config (org.apache.kafka.connect.runtime.Worker:462)
connect_1              | [2021-11-30 23:34:32,316] INFO Initializing: org.apache.kafka.connect.runtime.TransformationChain{} (org.apache.kafka.connect.runtime.Worker:516)
connect_1              | [2021-11-30 23:34:32,321] INFO ProducerConfig values:
connect_1              |    acks = -1
connect_1              |    batch.size = 16384
connect_1              |    bootstrap.servers = [kafka:19092]
connect_1              |    buffer.memory = 33554432
connect_1              |    client.dns.lookup = default
connect_1              |    client.id = connector-producer-redis-config-0
connect_1              |    compression.type = none
connect_1              |    connections.max.idle.ms = 540000
connect_1              |    delivery.timeout.ms = 2147483647
connect_1              |    enable.idempotence = false
connect_1              |    interceptor.classes = []
connect_1              |    key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
connect_1              |    linger.ms = 0
connect_1              |    max.block.ms = 9223372036854775807
connect_1              |    max.in.flight.requests.per.connection = 1
connect_1              |    max.request.size = 1048576
connect_1              |    metadata.max.age.ms = 300000
connect_1              |    metadata.max.idle.ms = 300000
connect_1              |    metric.reporters = []
connect_1              |    metrics.num.samples = 2
connect_1              |    metrics.recording.level = INFO
connect_1              |    metrics.sample.window.ms = 30000
connect_1              |    partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
connect_1              |    receive.buffer.bytes = 32768
connect_1              |    reconnect.backoff.max.ms = 1000
connect_1              |    reconnect.backoff.ms = 50
connect_1              |    request.timeout.ms = 2147483647
connect_1              |    retries = 2147483647
connect_1              |    retry.backoff.ms = 100
connect_1              |    sasl.client.callback.handler.class = null
connect_1              |    sasl.jaas.config = null
connect_1              |    sasl.kerberos.kinit.cmd = /usr/bin/kinit
connect_1              |    sasl.kerberos.min.time.before.relogin = 60000
connect_1              |    sasl.kerberos.service.name = null
connect_1              |    sasl.kerberos.ticket.renew.jitter = 0.05
connect_1              |    sasl.kerberos.ticket.renew.window.factor = 0.8
connect_1              |    sasl.login.callback.handler.class = null
connect_1              |    sasl.login.class = null
connect_1              |    sasl.login.refresh.buffer.seconds = 300
connect_1              |    sasl.login.refresh.min.period.seconds = 60
connect_1              |    sasl.login.refresh.window.factor = 0.8
connect_1              |    sasl.login.refresh.window.jitter = 0.05
connect_1              |    sasl.mechanism = GSSAPI
connect_1              |    security.protocol = PLAINTEXT
connect_1              |    security.providers = null
connect_1              |    send.buffer.bytes = 131072
connect_1              |    ssl.cipher.suites = null
connect_1              |    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
connect_1              |    ssl.endpoint.identification.algorithm = https
connect_1              |    ssl.key.password = null
connect_1              |    ssl.keymanager.algorithm = SunX509
connect_1              |    ssl.keystore.location = null
connect_1              |    ssl.keystore.password = null
connect_1              |    ssl.keystore.type = JKS
connect_1              |    ssl.protocol = TLS
connect_1              |    ssl.provider = null
connect_1              |    ssl.secure.random.implementation = null
connect_1              |    ssl.trustmanager.algorithm = PKIX
connect_1              |    ssl.truststore.location = null
connect_1              |    ssl.truststore.password = null
connect_1              |    ssl.truststore.type = JKS
connect_1              |    transaction.timeout.ms = 60000
connect_1              |    transactional.id = null
connect_1              |    value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
connect_1              |  (org.apache.kafka.clients.producer.ProducerConfig:347)
connect_1              | [2021-11-30 23:34:32,341] INFO Kafka version: 5.5.1-ccs (org.apache.kafka.common.utils.AppInfoParser:117)
connect_1              | [2021-11-30 23:34:32,341] INFO Kafka commitId: 3c4783aac9e33249 (org.apache.kafka.common.utils.AppInfoParser:118)
connect_1              | [2021-11-30 23:34:32,341] INFO Kafka startTimeMs: 1638315272341 (org.apache.kafka.common.utils.AppInfoParser:119)
connect_1              | [2021-11-30 23:34:32,351] INFO Created connector redis-config (org.apache.kafka.connect.cli.ConnectStandalone:112)
connect_1              | [2021-11-30 23:34:32,352] INFO Memory checker started. (org.apache.kafka.connect.redis.MemoryChecker:61)
connect_1              | [2021-11-30 23:34:32,354] INFO [Producer clientId=connector-producer-redis-config-0] Cluster ID: 7Rv27TYRSA2ZLyiWx0JFQA (org.apache.kafka.clients.Metadata:277)
connect_1              | [2021-11-30 23:34:32,515] INFO WorkerSourceTask{id=redis-config-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:216)
connect_1              | [2021-11-30 23:34:32,523] INFO Connected to redis-server[redis:6379] (com.moilioncircle.redis.replicator.RedisSocketReplicator:298)
connect_1              | [2021-11-30 23:34:32,523] INFO PING (com.moilioncircle.redis.replicator.RedisSocketReplicator:190)
connect_1              | [2021-11-30 23:34:32,524] INFO PONG (com.moilioncircle.redis.replicator.RedisSocketReplicator:193)
connect_1              | [2021-11-30 23:34:32,525] INFO REPLCONF listening-port 43202 (com.moilioncircle.redis.replicator.RedisSocketReplicator:202)
connect_1              | [2021-11-30 23:34:32,525] INFO OK (com.moilioncircle.redis.replicator.RedisSocketReplicator:205)
connect_1              | [2021-11-30 23:34:32,525] INFO REPLCONF ip-address 172.29.0.10 (com.moilioncircle.redis.replicator.RedisSocketReplicator:212)
connect_1              | [2021-11-30 23:34:32,525] INFO OK (com.moilioncircle.redis.replicator.RedisSocketReplicator:215)
connect_1              | [2021-11-30 23:34:32,525] INFO REPLCONF capa eof (com.moilioncircle.redis.replicator.RedisSocketReplicator:223)
connect_1              | [2021-11-30 23:34:32,526] INFO OK (com.moilioncircle.redis.replicator.RedisSocketReplicator:226)
connect_1              | [2021-11-30 23:34:32,526] INFO REPLCONF capa psync2 (com.moilioncircle.redis.replicator.RedisSocketReplicator:223)
connect_1              | [2021-11-30 23:34:32,526] INFO OK (com.moilioncircle.redis.replicator.RedisSocketReplicator:226)
connect_1              | [2021-11-30 23:34:32,526] INFO PSYNC cabc765f5f9bd6a0fab5b8708ead10c08ac60080 0 (com.moilioncircle.redis.replicator.RedisSocketReplicator:360)
connect_1              | [2021-11-30 23:34:32,527] INFO FULLRESYNC dbd3cbc4b2e3602ecac463f7820a49ee109d2a6c 0 (com.moilioncircle.redis.replicator.RedisSocketReplicator:116)
connect_1              | [2021-11-30 23:34:32,663] INFO RDB dump file size:11809 (com.moilioncircle.redis.replicator.RedisSocketReplicator:144)
connect_1              | [2021-11-30 23:34:32,666] INFO RDB redis-ver: 6.0.6 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,667] INFO RDB redis-bits: 64 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,667] INFO RDB ctime: 1638315272 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,667] INFO RDB used-mem: 1977184 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,667] INFO RDB repl-stream-db: 0 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,668] INFO RDB repl-id: dbd3cbc4b2e3602ecac463f7820a49ee109d2a6c (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,668] INFO RDB repl-offset: 0 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,668] INFO RDB aof-preamble: 0 (com.moilioncircle.redis.replicator.rdb.DefaultRdbVisitor:150)
connect_1              | [2021-11-30 23:34:32,674] INFO heartbeat started. (com.moilioncircle.redis.replicator.RedisSocketReplicator:239)
connect_1              | [2021-11-30 23:34:32,681] ERROR Error converting event to JSON (org.apache.kafka.connect.redis.RedisSourceTask:102)
connect_1              | com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.moilioncircle.redis.replicator.event.PreRdbSyncEvent and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
connect_1              |    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
connect_1              |    at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191)
connect_1              |    at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:404)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:71)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:33)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:4094)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3404)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.getSourceRecord(RedisSourceTask.java:99)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.poll(RedisSourceTask.java:77)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:272)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:239)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
connect_1              |    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
connect_1              |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
connect_1              |    at java.lang.Thread.run(Thread.java:748)
connect_1              | [2021-11-30 23:34:32,791] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 3 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:32,895] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 4 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,003] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 5 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,110] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 6 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,216] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 7 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,323] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 8 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,430] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 9 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,536] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 10 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,642] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 11 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,746] WARN [Producer clientId=connector-producer-redis-config-0] Error while fetching metadata with correlation id 12 : {redis-server=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:1077)
connect_1              | [2021-11-30 23:34:33,907] ERROR Error converting event to JSON (org.apache.kafka.connect.redis.RedisSourceTask:102)
connect_1              | com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.moilioncircle.redis.replicator.event.PreCommandSyncEvent and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
connect_1              |    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
connect_1              |    at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191)
connect_1              |    at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:404)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:71)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:33)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:4094)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3404)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.getSourceRecord(RedisSourceTask.java:99)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.poll(RedisSourceTask.java:77)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:272)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:239)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
connect_1              |    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
connect_1              |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
connect_1              |    at java.lang.Thread.run(Thread.java:748)
connect_1              | [2021-11-30 23:35:32,350] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
connect_1              | [2021-11-30 23:35:32,351] INFO WorkerSourceTask{id=redis-config-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
connect_1              | [2021-11-30 23:35:32,372] INFO WorkerSourceTask{id=redis-config-0} Finished commitOffsets successfully in 21 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:525)
connect_1              | [2021-11-30 23:36:32,372] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
connect_1              | [2021-11-30 23:36:32,373] INFO WorkerSourceTask{id=redis-config-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
connect_1              | [2021-11-30 23:36:32,374] INFO WorkerSourceTask{id=redis-config-0} Finished commitOffsets successfully in 1 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:525)
connect_1              | [2021-11-30 23:37:32,375] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
connect_1              | [2021-11-30 23:37:32,375] INFO WorkerSourceTask{id=redis-config-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
connect_1              | [2021-11-30 23:37:32,376] INFO WorkerSourceTask{id=redis-config-0} Finished commitOffsets successfully in 1 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:525)
connect_1              | [2021-11-30 23:38:32,377] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
connect_1              | [2021-11-30 23:38:32,377] INFO WorkerSourceTask{id=redis-config-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
connect_1              | [2021-11-30 23:38:32,378] INFO WorkerSourceTask{id=redis-config-0} Finished commitOffsets successfully in 1 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:525)
connect_1              | [2021-11-30 23:39:32,379] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
connect_1              | [2021-11-30 23:39:32,379] INFO WorkerSourceTask{id=redis-config-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
connect_1              | [2021-11-30 23:39:32,381] INFO WorkerSourceTask{id=redis-config-0} Finished commitOffsets successfully in 2 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:525)

Looks like it has something related with disable SerializationFeature.FAIL_ON_EMPTY_BEANS but after searching, I think it should be fixed in java code level.

How can I fix it?

Note

scmurdock commented 2 years ago

The kafka service is not listening. This sometimes happens the first time you start up the docker compose file. Try doing 'docker-compose down' and then 'docker-compose up' again to try again. The first time you start it takes quite a while to download all the images before running, and there can be such a delay that it prevents proper startup. Hope that helps.

On Tue, Nov 30, 2021 at 4:44 PM JeongWoo Choi @.***> wrote:

ENV

  • OS: Ubuntu 20.04.2 LTS
  • Docker version: Docker version 20.10.6, build 370c289
  • docker-compose version: docker-compose version 1.29.1, build c34c88b2

Problem I've cloned the repo and run docker-compose up, all containers are healthy except the connect service, which uses gcr.io/simulation-images/kafka-connect-redis-source image:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

fbd12e6e5264 gcr.io/simulation-images/kafka-connect-redis-source "/bin/sh -c 'connect…" 3 minutes ago Up 3 minutes (health: starting) 0.0.0.0:5005->5005/tcp, :::5005->5005/tcp, 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp nd029-c2-apache-spark-and-spark-streaming-starter_connect_1

f4b45d4f2930 gcr.io/simulation-images/trucking-simulation "java -jar /trucking…" 3 minutes ago Up 3 minutes 4567/tcp nd029-c2-apache-spark-and-spark-streaming-starter_trucking-simulation_1

1ccf3fc356d1 gcr.io/simulation-images/banking-simulation "java -jar /banking.…" 3 minutes ago Up 3 minutes 4567/tcp nd029-c2-apache-spark-and-spark-streaming-starter_banking-simulation_1

c2a2ca10f99b gcr.io/simulation-images/stedi "java -jar /stedi.jar" 3 minutes ago Up 3 minutes 0.0.0.0:4567->4567/tcp, :::4567->4567/tcp nd029-c2-apache-spark-and-spark-streaming-starter_stedi_1

f3f3d94276e3 confluentinc/cp-kafka:5.5.1 "/etc/confluent/dock…" 3 minutes ago Up 3 minutes 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp nd029-c2-apache-spark-and-spark-streaming-starter_kafka_1

27a7748a170a confluentinc/cp-zookeeper:5.5.1 "/etc/confluent/dock…" 3 minutes ago Up 3 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp nd029-c2-apache-spark-and-spark-streaming-starter_zookeeper_1

62f27d778172 redis:6.0.6 "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp nd029-c2-apache-spark-and-spark-streaming-starter_redis_1

c57142f3f12f bitnami/spark:3-debian-10 "/opt/bitnami/script…" 3 minutes ago Up 3 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp nd029-c2-apache-spark-and-spark-streaming-starter_spark_1

It kept restarting, so I checked the log of that container:

$ docker-compose logs connect

Attaching to nd029-c2-apache-spark-and-spark-streaming-starter_connect_1

connect_1 | [2021-11-30 23:34:27,045] INFO Kafka Connect standalone worker initializing ... (org.apache.kafka.connect.cli.ConnectStandalone:69)

connect_1 | [2021-11-30 23:34:27,054] INFO WorkerInfo values:

connect_1 | jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/log/kafka, -Dlog4j.configuration=file:/etc/kafka/connect-log4j.properties

connect_1 | jvm.spec = Azul Systems, Inc., OpenJDK 64-Bit Server VM, 1.8.0_212, 25.212-b04

connect_1 | jvm.classpath = /usr/share/java/kafka-connect-redis-source/commons-logging-1.2.jar:/usr/share/java/kafka-connect-redis-source/jedis-2.9.0.jar:/usr/share/java/kafka-connect-redis-source/kafka-connect-redis-1.0-SNAPSHOT.jar:/usr/share/java/kafka-connect-redis-source/redis-replicator-3.0.1.jar:/usr/share/java/kafka/activation-1.1.1.jar:/usr/share/java/kafka/aopalliance-repackaged-2.5.0.jar:/usr/share/java/kafka/argparse4j-0.7.0.jar:/usr/share/java/kafka/audience-annotations-0.5.0.jar:/usr/share/java/kafka/avro-1.9.2.jar:/usr/share/java/kafka/commons-cli-1.4.jar:/usr/share/java/kafka/commons-codec-1.11.jar:/usr/share/java/kafka/commons-compress-1.19.jar:/usr/share/java/kafka/commons-lang3-3.8.1.jar:/usr/share/java/kafka/commons-logging-1.2.jar:/usr/share/java/kafka/connect-api-5.5.1-ccs.jar:/usr/share/java/kafka/connect-basic-auth-extension-5.5.1-ccs.jar:/usr/share/java/kafka/connect-file-5.5.1-ccs.jar:/usr/share/java/kafka/connect-json-5.5.1-ccs.jar:/usr/share/java/kafka/connect-mirror-5.5.1-ccs.jar:/usr/share/java/kafka/connect-mirror-client-5.5.1-ccs.jar:/usr/share/java/kafka/connect-runtime-5.5.1-ccs.jar:/usr/share/java/kafka/connect-transforms-5.5.1-ccs.jar:/usr/share/java/kafka/hk2-api-2.5.0.jar:/usr/share/java/kafka/hk2-locator-2.5.0.jar:/usr/share/java/kafka/hk2-utils-2.5.0.jar:/usr/share/java/kafka/httpclient-4.5.11.jar:/usr/share/java/kafka/httpcore-4.4.13.jar:/usr/share/java/kafka/httpmime-4.5.11.jar:/usr/share/java/kafka/jackson-annotations-2.10.2.jar:/usr/share/java/kafka/jackson-core-2.10.2.jar:/usr/share/java/kafka/jackson-databind-2.10.2.jar:/usr/share/java/kafka/jackson-dataformat-csv-2.10.2.jar:/usr/share/java/kafka/jackson-datatype-jdk8-2.10.2.jar:/usr/share/java/kafka/jackson-jaxrs-base-2.10.2.jar:/usr/share/java/kafka/jackson-jaxrs-json-provider-2.10.2.jar:/usr/share/java/kafka/jackson-module-jaxb-annotations-2.10.2.jar:/usr/share/java/kafka/jackson-module-paranamer-2.10.2.jar:/usr/share/java/kafka/jackson-module-scala_2.12-2.10.2.jar:/usr/share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/share/java/kafka/jakarta.annotation-api-1.3.4.jar:/usr/share/java/kafka/jakarta.inject-2.5.0.jar:/usr/share/java/kafka/jakarta.ws.rs-api-2.1.5.jar:/usr/share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/share/java/kafka/javassist-3.22.0-CR2.jar:/usr/share/java/kafka/javassist-3.26.0-GA.jar:/usr/share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/share/java/kafka/jaxb-api-2.3.0.jar:/usr/share/java/kafka/jersey-client-2.28.jar:/usr/share/java/kafka/jersey-common-2.28.jar:/usr/share/java/kafka/jersey-container-servlet-2.28.jar:/usr/share/java/kafka/jersey-container-servlet-core-2.28.jar:/usr/share/java/kafka/jersey-hk2-2.28.jar:/usr/share/java/kafka/jersey-media-jaxb-2.28.jar:/usr/share/java/kafka/jersey-server-2.28.jar:/usr/share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/share/java/kafka/jopt-simple-5.0.4.jar:/usr/share/java/kafka/kafka-clients-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-log4j-appender-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-examples-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-scala_2.12-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-streams-test-utils-5.5.1-ccs.jar:/usr/share/java/kafka/kafka-tools-5.5.1-ccs.jar:/usr/share/java/kafka/kafka.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-javadoc.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-scaladoc.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-sources.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-test-sources.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs-test.jar:/usr/share/java/kafka/kafka_2.12-5.5.1-ccs.jar:/usr/share/java/kafka/log4j-1.2.17.jar:/usr/share/java/kafka/lz4-java-1.7.1.jar:/usr/share/java/kafka/maven-artifact-3.6.3.jar:/usr/share/java/kafka/metrics-core-2.2.0.jar:/usr/share/java/kafka/netty-buffer-4.1.48.Final.jar:/usr/share/java/kafka/netty-codec-4.1.48.Final.jar:/usr/share/java/kafka/netty-common-4.1.48.Final.jar:/usr/share/java/kafka/netty-handler-4.1.48.Final.jar:/usr/share/java/kafka/netty-resolver-4.1.48.Final.jar:/usr/share/java/kafka/netty-transport-4.1.48.Final.jar:/usr/share/java/kafka/netty-transport-native-epoll-4.1.48.Final.jar:/usr/share/java/kafka/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/share/java/kafka/paranamer-2.8.jar:/usr/share/java/kafka/plexus-utils-3.2.1.jar:/usr/share/java/kafka/reflections-0.9.12.jar:/usr/share/java/kafka/rocksdbjni-5.18.3.jar:/usr/share/java/kafka/scala-collection-compat_2.12-2.1.3.jar:/usr/share/java/kafka/scala-java8-compat_2.12-0.9.0.jar:/usr/share/java/kafka/scala-library-2.12.10.jar:/usr/share/java/kafka/scala-logging_2.12-3.9.2.jar:/usr/share/java/kafka/scala-reflect-2.12.10.jar:/usr/share/java/kafka/slf4j-api-1.7.30.jar:/usr/share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/share/java/kafka/snappy-java-1.1.7.3.jar:/usr/share/java/kafka/support-metrics-client-5.5.1-ccs.jar:/usr/share/java/kafka/support-metrics-common-5.5.1-ccs.jar:/usr/share/java/kafka/validation-api-2.0.1.Final.jar:/usr/share/java/kafka/zookeeper-3.5.8.jar:/usr/share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/share/java/kafka/confluent-metrics-5.5.1-ce.jar:/usr/share/java/confluent-common/build-tools-5.5.1.jar:/usr/share/java/confluent-common/common-config-5.5.1.jar:/usr/share/java/confluent-common/common-metrics-5.5.1.jar:/usr/share/java/confluent-common/common-utils-5.5.1.jar:/usr/share/java/confluent-common/slf4j-api-1.7.26.jar:/usr/share/java/kafka-serde-tools/animal-sniffer-annotations-1.14.jar:/usr/share/java/kafka-serde-tools/annotations-13.0.jar:/usr/share/java/kafka-serde-tools/avro-1.9.2.jar:/usr/share/java/kafka-serde-tools/checker-compat-qual-2.0.0.jar:/usr/share/java/kafka-serde-tools/classgraph-4.8.21.jar:/usr/share/java/kafka-serde-tools/commons-collections-3.2.2.jar:/usr/share/java/kafka-serde-tools/commons-compress-1.19.jar:/usr/share/java/kafka-serde-tools/commons-digester-1.8.1.jar:/usr/share/java/kafka-serde-tools/commons-logging-1.2.jar:/usr/share/java/kafka-serde-tools/commons-validator-1.6.jar:/usr/share/java/kafka-serde-tools/error_prone_annotations-2.3.4.jar:/usr/share/java/kafka-serde-tools/gson-2.8.5.jar:/usr/share/java/kafka-serde-tools/guava-24.0-jre.jar:/usr/share/java/kafka-serde-tools/handy-uri-templates-2.1.8.jar:/usr/share/java/kafka-serde-tools/j2objc-annotations-1.1.jar:/usr/share/java/kafka-serde-tools/jackson-annotations-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-core-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-databind-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-guava-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-jdk8-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-joda-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-datatype-jsr310-2.10.2.jar:/usr/share/java/kafka-serde-tools/jackson-module-parameter-names-2.10.2.jar:/usr/share/java/kafka-serde-tools/jakarta.annotation-api-1.3.5.jar:/usr/share/java/kafka-serde-tools/jakarta.inject-2.6.1.jar:/usr/share/java/kafka-serde-tools/jakarta.ws.rs-api-2.1.6.jar:/usr/share/java/kafka-serde-tools/javax.ws.rs-api-2.1.1.jar:/usr/share/java/kafka-serde-tools/jersey-common-2.30.jar:/usr/share/java/kafka-serde-tools/joda-time-2.9.9.jar:/usr/share/java/kafka-serde-tools/json-20190722.jar:/usr/share/java/kafka-serde-tools/jsr305-1.3.9.jar:/usr/share/java/kafka-serde-tools/kafka-avro-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-avro-converter-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-avro-data-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-json-schema-converter-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-protobuf-converter-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-json-schema-provider-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-json-schema-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-json-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-protobuf-provider-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-protobuf-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-schema-registry-client-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-schema-serializer-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-streams-5.5.1-ccs.jar:/usr/share/java/kafka-serde-tools/kafka-streams-avro-serde-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-streams-json-schema-serde-5.5.1.jar:/usr/share/java/kafka-serde-tools/kafka-streams-protobuf-serde-5.5.1.jar:/usr/share/java/kafka-serde-tools/kotlin-reflect-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-script-runtime-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-common-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-compiler-embeddable-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-compiler-impl-embeddable-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-scripting-jvm-1.3.50.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-common-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-jdk7-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlin-stdlib-jdk8-1.3.71.jar:/usr/share/java/kafka-serde-tools/kotlinx-coroutines-core-1.1.1.jar:/usr/share/java/kafka-serde-tools/kotlinx-coroutines-core-common-1.1.1.jar:/usr/share/java/kafka-serde-tools/mbknor-jackson-jsonschema_2.12-1.0.39.jar:/usr/share/java/kafka-serde-tools/okio-2.5.0.jar:/usr/share/java/kafka-serde-tools/org.everit.json.schema-1.12.1.jar:/usr/share/java/kafka-serde-tools/osgi-resource-locator-1.0.3.jar:/usr/share/java/kafka-serde-tools/protobuf-java-3.11.4.jar:/usr/share/java/kafka-serde-tools/protobuf-java-util-3.11.4.jar:/usr/share/java/kafka-serde-tools/re2j-1.3.jar:/usr/share/java/kafka-serde-tools/rocksdbjni-5.18.3.jar:/usr/share/java/kafka-serde-tools/scala-library-2.12.10.jar:/usr/share/java/kafka-serde-tools/swagger-annotations-1.6.0.jar:/usr/share/java/kafka-serde-tools/validation-api-2.0.1.Final.jar:/usr/share/java/kafka-serde-tools/wire-runtime-3.2.2.jar:/usr/share/java/kafka-serde-tools/wire-schema-3.2.2.jar:/usr/share/java/monitoring-interceptors/monitoring-interceptors-5.5.1.jar:/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.5.0.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/avro-1.9.2.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-codec-1.11.jar:/usr/bin/../share/java/kafka/commons-compress-1.19.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/commons-logging-1.2.jar:/usr/bin/../share/java/kafka/connect-api-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-file-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-json-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.5.0.jar:/usr/bin/../share/java/kafka/hk2-locator-2.5.0.jar:/usr/bin/../share/java/kafka/hk2-utils-2.5.0.jar:/usr/bin/../share/java/kafka/httpclient-4.5.11.jar:/usr/bin/../share/java/kafka/httpcore-4.4.13.jar:/usr/bin/../share/java/kafka/httpmime-4.5.11.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-core-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-databind-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-module-paranamer-2.10.2.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.12-2.10.2.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.4.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.5.0.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.5.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/bin/../share/java/kafka/javassist-3.22.0-CR2.jar:/usr/bin/../share/java/kafka/javassist-3.26.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.28.jar:/usr/bin/../share/java/kafka/jersey-common-2.28.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.28.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.28.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.28.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.28.jar:/usr/bin/../share/java/kafka/jersey-server-2.28.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.12-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-javadoc.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-scaladoc.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-sources.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-test-sources.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs-test.jar:/usr/bin/../share/java/kafka/kafka_2.12-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/log4j-1.2.17.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.6.3.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.48.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.18.3.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.12-2.1.3.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.12-0.9.0.jar:/usr/bin/../share/java/kafka/scala-library-2.12.10.jar:/usr/bin/../share/java/kafka/scala-logging_2.12-3.9.2.jar:/usr/bin/../share/java/kafka/scala-reflect-2.12.10.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.30.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.7.3.jar:/usr/bin/../share/java/kafka/support-metrics-client-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/support-metrics-common-5.5.1-ccs.jar:/usr/bin/../share/java/kafka/validation-api-2.0.1.Final.jar:/usr/bin/../share/java/kafka/zookeeper-3.5.8.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/bin/../share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/bin/../share/java/kafka/confluent-metrics-5.5.1-ce.jar:/usr/bin/../support-metrics-client/build/dependant-libs-2.12/:/usr/bin/../support-metrics-client/build/libs/:/usr/share/java/support-metrics-client/*

connect_1 | os.spec = Linux, amd64, 5.11.0-38-generic

connect_1 | os.vcpus = 16

connect_1 | (org.apache.kafka.connect.runtime.WorkerInfo:71)

connect_1 | [2021-11-30 23:34:27,062] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectStandalone:78)

connect_1 | [2021-11-30 23:34:29,761] INFO Registered loader: @.*** (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:262)

connect_1 | [2021-11-30 23:34:29,761] INFO Added plugin 'org.apache.kafka.connect.redis.RedisSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,762] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,762] INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,762] INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,769] INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,772] INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,772] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,772] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,773] INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,774] INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,774] INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'io.confluent.connect.json.JsonSchemaConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'io.confluent.connect.protobuf.ProtobufConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,775] INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,776] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,777] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,778] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,779] INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,780] INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:191)

connect_1 | [2021-11-30 23:34:29,782] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,782] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,783] INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,783] INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,783] INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,783] INFO Added aliases 'RedisSourceConnector' and 'RedisSource' to plugin 'org.apache.kafka.connect.redis.RedisSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,783] INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,784] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,784] INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,784] INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,784] INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,784] INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,784] INFO Added aliases 'AvroConverter' and 'Avro' to plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'JsonSchemaConverter' and 'JsonSchema' to plugin 'io.confluent.connect.json.JsonSchemaConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'ProtobufConverter' and 'Protobuf' to plugin 'io.confluent.connect.protobuf.ProtobufConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,785] INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,786] INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,787] INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,787] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,787] INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)

connect_1 | [2021-11-30 23:34:29,787] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,788] INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)

connect_1 | [2021-11-30 23:34:29,788] INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)

connect_1 | [2021-11-30 23:34:29,788] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)

connect_1 | [2021-11-30 23:34:29,788] INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:416)

connect_1 | [2021-11-30 23:34:29,788] INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,789] INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,789] INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:419)

connect_1 | [2021-11-30 23:34:29,806] INFO StandaloneConfig values:

connect_1 | access.control.allow.methods =

connect_1 | access.control.allow.origin =

connect_1 | admin.listeners = null

connect_1 | bootstrap.servers = [kafka:19092]

connect_1 | client.dns.lookup = default

connect_1 | config.providers = []

connect_1 | connector.client.config.override.policy = None

connect_1 | header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter

connect_1 | internal.key.converter = class org.apache.kafka.connect.json.JsonConverter

connect_1 | internal.value.converter = class org.apache.kafka.connect.json.JsonConverter

connect_1 | key.converter = class org.apache.kafka.connect.converters.ByteArrayConverter

connect_1 | listeners = null

connect_1 | metric.reporters = []

connect_1 | metrics.num.samples = 2

connect_1 | metrics.recording.level = INFO

connect_1 | metrics.sample.window.ms = 30000

connect_1 | offset.flush.interval.ms = 60000

connect_1 | offset.flush.timeout.ms = 5000

connect_1 | offset.storage.file.filename = /tmp/connect.offsets

connect_1 | plugin.path = [share/java, /opt/confluent-5.1.0/share/confluent-hub-components]

connect_1 | rest.advertised.host.name = null

connect_1 | rest.advertised.listener = null

connect_1 | rest.advertised.port = null

connect_1 | rest.extension.classes = []

connect_1 | rest.host.name = null

connect_1 | rest.port = 8083

connect_1 | ssl.cipher.suites = null

connect_1 | ssl.client.auth = none

connect_1 | ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

connect_1 | ssl.endpoint.identification.algorithm = https

connect_1 | ssl.key.password = null

connect_1 | ssl.keymanager.algorithm = SunX509

connect_1 | ssl.keystore.location = null

connect_1 | ssl.keystore.password = null

connect_1 | ssl.keystore.type = JKS

connect_1 | ssl.protocol = TLS

connect_1 | ssl.provider = null

connect_1 | ssl.secure.random.implementation = null

connect_1 | ssl.trustmanager.algorithm = PKIX

connect_1 | ssl.truststore.location = null

connect_1 | ssl.truststore.password = null

connect_1 | ssl.truststore.type = JKS

connect_1 | task.shutdown.graceful.timeout.ms = 5000

connect_1 | topic.tracking.allow.reset = true

connect_1 | topic.tracking.enable = true

connect_1 | value.converter = class org.apache.kafka.connect.storage.StringConverter

connect_1 | (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:347)

connect_1 | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)

connect_1 | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.key.converter.schemas.enable' (along with all configuration for 'internal.key.converter') is deprecated and may be removed in an upcoming release. The specified value 'false' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)

connect_1 | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)

connect_1 | [2021-11-30 23:34:29,807] INFO Worker configuration property 'internal.value.converter.schemas.enable' (along with all configuration for 'internal.value.converter') is deprecated and may be removed in an upcoming release. The specified value 'false' matches the default, so this property can be safely removed from the worker configuration. (org.apache.kafka.connect.runtime.WorkerConfig:363)

connect_1 | [2021-11-30 23:34:29,808] INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils:43)

connect_1 | [2021-11-30 23:34:29,811] INFO AdminClientConfig values:

connect_1 | bootstrap.servers = [kafka:19092]

connect_1 | client.dns.lookup = default

connect_1 | client.id =

connect_1 | connections.max.idle.ms = 300000

connect_1 | default.api.timeout.ms = 60000

connect_1 | metadata.max.age.ms = 300000

connect_1 | metric.reporters = []

connect_1 | metrics.num.samples = 2

connect_1 | metrics.recording.level = INFO

connect_1 | metrics.sample.window.ms = 30000

connect_1 | receive.buffer.bytes = 65536

connect_1 | reconnect.backoff.max.ms = 1000

connect_1 | reconnect.backoff.ms = 50

connect_1 | request.timeout.ms = 30000

connect_1 | retries = 2147483647

connect_1 | retry.backoff.ms = 100

connect_1 | sasl.client.callback.handler.class = null

connect_1 | sasl.jaas.config = null

connect_1 | sasl.kerberos.kinit.cmd = /usr/bin/kinit

connect_1 | sasl.kerberos.min.time.before.relogin = 60000

connect_1 | sasl.kerberos.service.name = null

connect_1 | sasl.kerberos.ticket.renew.jitter = 0.05

connect_1 | sasl.kerberos.ticket.renew.window.factor = 0.8

connect_1 | sasl.login.callback.handler.class = null

connect_1 | sasl.login.class = null

connect_1 | sasl.login.refresh.buffer.seconds = 300

connect_1 | sasl.login.refresh.min.period.seconds = 60

connect_1 | sasl.login.refresh.window.factor = 0.8

connect_1 | sasl.login.refresh.window.jitter = 0.05

connect_1 | sasl.mechanism = GSSAPI

connect_1 | security.protocol = PLAINTEXT

connect_1 | security.providers = null

connect_1 | send.buffer.bytes = 131072

connect_1 | ssl.cipher.suites = null

connect_1 | ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

connect_1 | ssl.endpoint.identification.algorithm = https

connect_1 | ssl.key.password = null

connect_1 | ssl.keymanager.algorithm = SunX509

connect_1 | ssl.keystore.location = null

connect_1 | ssl.keystore.password = null

connect_1 | ssl.keystore.type = JKS

connect_1 | ssl.protocol = TLS

connect_1 | ssl.provider = null

connect_1 | ssl.secure.random.implementation = null

connect_1 | ssl.trustmanager.algorithm = PKIX

connect_1 | ssl.truststore.location = null

connect_1 | ssl.truststore.password = null

connect_1 | ssl.truststore.type = JKS

connect_1 | (org.apache.kafka.clients.admin.AdminClientConfig:347)

connect_1 | [2021-11-30 23:34:29,860] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,860] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,860] WARN The configuration 'max.request.size' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,860] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,860] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,860] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,861] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,861] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,861] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,861] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,861] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)

connect_1 | [2021-11-30 23:34:29,861] INFO Kafka version: 5.5.1-ccs (org.apache.kafka.common.utils.AppInfoParser:117)

connect_1 | [2021-11-30 23:34:29,861] INFO Kafka commitId: 3c4783aac9e33249 (org.apache.kafka.common.utils.AppInfoParser:118)

connect_1 | [2021-11-30 23:34:29,861] INFO Kafka startTimeMs: 1638315269861 (org.apache.kafka.common.utils.AppInfoParser:119)

connect_1 | [2021-11-30 23:34:29,879] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)

connect_1 | [2021-11-30 23:34:29,981] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)

connect_1 | [2021-11-30 23:34:30,083] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)

connect_1 | [2021-11-30 23:34:30,385] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.6:19092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:763)

connect_1 | [2021-11-30 23:34:31,212] INFO Kafka cluster ID: 7Rv27TYRSA2ZLyiWx0JFQA (org.apache.kafka.connect.util.ConnectUtils:59)

connect_1 | [2021-11-30 23:34:31,236] INFO Logging initialized @4693ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:169)

connect_1 | [2021-11-30 23:34:31,300] INFO Added connector for http://:8083 (org.apache.kafka.connect.runtime.rest.RestServer:131)

connect_1 | [2021-11-30 23:34:31,301] INFO Initializing REST server (org.apache.kafka.connect.runtime.rest.RestServer:203)

connect_1 | [2021-11-30 23:34:31,310] INFO jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 1.8.0_212-b04 (org.eclipse.jetty.server.Server:359)

connect_1 | [2021-11-30 23:34:31,344] INFO Started @.***{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:330)

connect_1 | [2021-11-30 23:34:31,344] INFO Started @4802ms (org.eclipse.jetty.server.Server:399)

connect_1 | [2021-11-30 23:34:31,370] INFO Advertised URI: http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:365)

connect_1 | [2021-11-30 23:34:31,371] INFO REST server listening at http://172.29.0.10:8083/, advertising URL http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:218)

connect_1 | [2021-11-30 23:34:31,372] INFO Advertised URI: http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:365)

connect_1 | [2021-11-30 23:34:31,374] INFO REST admin endpoints at http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:219)

connect_1 | [2021-11-30 23:34:31,375] INFO Advertised URI: http://172.29.0.10:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:365)

connect_1 | [2021-11-30 23:34:31,375] INFO Setting up None Policy for ConnectorClientConfigOverride. This will disallow any client configuration to be overridden (org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy:45)

connect_1 | [2021-11-30 23:34:31,393] INFO Kafka version: 5.5.1-ccs (org.apache.kafka.common.utils.AppInfoParser:117)

connect_1 | [2021-11-30 23:34:31,393] INFO Kafka commitId: 3c4783aac9e33249 (org.apache.kafka.common.utils.AppInfoParser:118)

connect_1 | [2021-11-30 23:34:31,394] INFO Kafka startTimeMs: 1638315271393 (org.apache.kafka.common.utils.AppInfoParser:119)

connect_1 | [2021-11-30 23:34:31,581] INFO JsonConverterConfig values:

connect_1 | converter.type = key

connect_1 | decimal.format = BASE64

connect_1 | schemas.cache.size = 1000

connect_1 | schemas.enable = false

connect_1 | (org.apache.kafka.connect.json.JsonConverterConfig:347)

connect_1 | [2021-11-30 23:34:31,583] INFO JsonConverterConfig values:

connect_1 | converter.type = value

connect_1 | decimal.format = BASE64

connect_1 | schemas.cache.size = 1000

connect_1 | schemas.enable = false

connect_1 | (org.apache.kafka.connect.json.JsonConverterConfig:347)

connect_1 | [2021-11-30 23:34:31,597] INFO Kafka Connect standalone worker initialization took 4550ms (org.apache.kafka.connect.cli.ConnectStandalone:100)

connect_1 | [2021-11-30 23:34:31,598] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:51)

connect_1 | [2021-11-30 23:34:31,599] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:93)

connect_1 | [2021-11-30 23:34:31,599] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:184)

connect_1 | [2021-11-30 23:34:31,600] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:58)

connect_1 | [2021-11-30 23:34:31,604] INFO Worker started (org.apache.kafka.connect.runtime.Worker:191)

connect_1 | [2021-11-30 23:34:31,604] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:95)

connect_1 | [2021-11-30 23:34:31,605] INFO Initializing REST resources (org.apache.kafka.connect.runtime.rest.RestServer:223)

connect_1 | [2021-11-30 23:34:31,668] INFO Adding admin resources to main listener (org.apache.kafka.connect.runtime.rest.RestServer:240)

connect_1 | [2021-11-30 23:34:31,743] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session:333)

connect_1 | [2021-11-30 23:34:31,743] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session:338)

connect_1 | [2021-11-30 23:34:31,744] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session:140)

connect_1 | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime

connect_1 | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be ignored.

connect_1 | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime

connect_1 | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.RootResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.

connect_1 | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime

connect_1 | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.

connect_1 | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime

connect_1 | WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored.

connect_1 | Nov 30, 2021 11:34:32 PM org.glassfish.jersey.internal.Errors logErrors

connect_1 | WARNING: The following warnings have been detected: WARNING: The (sub)resource method listLoggers in org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains empty path annotation.

connect_1 | WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

connect_1 | WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

connect_1 | WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.

connect_1 | WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

connect_1 |

connect_1 | [2021-11-30 23:34:32,290] INFO Started @.***{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:825)

connect_1 | [2021-11-30 23:34:32,290] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:313)

connect_1 | [2021-11-30 23:34:32,290] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)

connect_1 | [2021-11-30 23:34:32,296] INFO AbstractConfig values:

connect_1 | (org.apache.kafka.common.config.AbstractConfig:347)

connect_1 | [2021-11-30 23:34:32,302] INFO ConnectorConfig values:

connect_1 | config.action.reload = restart

connect_1 | connector.class = org.apache.kafka.connect.redis.RedisSourceConnector

connect_1 | errors.log.enable = false

connect_1 | errors.log.include.messages = false

connect_1 | errors.retry.delay.max.ms = 60000

connect_1 | errors.retry.timeout = 0

connect_1 | errors.tolerance = none

connect_1 | header.converter = null

connect_1 | key.converter = null

connect_1 | name = redis-config

connect_1 | tasks.max = 1

connect_1 | transforms = []

connect_1 | value.converter = null

connect_1 | (org.apache.kafka.connect.runtime.ConnectorConfig:347)

connect_1 | [2021-11-30 23:34:32,303] INFO EnrichedConnectorConfig values:

connect_1 | config.action.reload = restart

connect_1 | connector.class = org.apache.kafka.connect.redis.RedisSourceConnector

connect_1 | errors.log.enable = false

connect_1 | errors.log.include.messages = false

connect_1 | errors.retry.delay.max.ms = 60000

connect_1 | errors.retry.timeout = 0

connect_1 | errors.tolerance = none

connect_1 | header.converter = null

connect_1 | key.converter = null

connect_1 | name = redis-config

connect_1 | tasks.max = 1

connect_1 | transforms = []

connect_1 | value.converter = null

connect_1 | (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)

connect_1 | [2021-11-30 23:34:32,303] INFO Creating connector redis-config of type org.apache.kafka.connect.redis.RedisSourceConnector (org.apache.kafka.connect.runtime.Worker:253)

connect_1 | [2021-11-30 23:34:32,305] INFO Instantiated connector redis-config with version 5.5.1-ccs of type class org.apache.kafka.connect.redis.RedisSourceConnector (org.apache.kafka.connect.runtime.Worker:256)

connect_1 | [2021-11-30 23:34:32,306] INFO Finished creating connector redis-config (org.apache.kafka.connect.runtime.Worker:275)

connect_1 | [2021-11-30 23:34:32,307] INFO SourceConnectorConfig values:

connect_1 | config.action.reload = restart

connect_1 | connector.class = org.apache.kafka.connect.redis.RedisSourceConnector

connect_1 | errors.log.enable = false

connect_1 | errors.log.include.messages = false

connect_1 | errors.retry.delay.max.ms = 60000

connect_1 | errors.retry.timeout = 0

connect_1 | errors.tolerance = none

connect_1 | header.converter = null

connect_1 |

rightx2 commented 2 years ago

@scmurdock hi, I've already tried restarting more than 20 times. I even prune all containers, images, volumes, and rebuild them, but still faced with the same problem. Have you tried docker-compose up?

scmurdock commented 2 years ago

Pruning will not fix it. Yes I have done it. What are the errors for the Kafka container?

On Tue, Nov 30, 2021, 6:26 PM JeongWoo Choi @.***> wrote:

@scmurdock https://github.com/scmurdock hi, I've already tried restarting more than 20 times. I even prune all containers, images, volumes, and rebuild them, but still faced with the same problem. Have you tried docker-compose up?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/udacity/nd029-c2-apache-spark-and-spark-streaming-starter/issues/1#issuecomment-983194284, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJBTBQKH67MORIVOCOUG3TUOV2T7ANCNFSM5JDCL7VA .

rightx2 commented 2 years ago

Here is the log (looks like no error exists in kafka?): https://gist.github.com/rightx2/09f173cf6cc8d48a39ee1872f1ba3282

rightx2 commented 2 years ago

@scmurdock Any idea ?

scmurdock commented 2 years ago

Thank you for sending the Kafka logs. Do you mind sending bother Kafka and connect logs for the same time period when the error occurs with connect?

Also, have you tried using the kafka-console-consumer command to consume the redis-server topic while running a simple redis set command?

On Wed, Dec 1, 2021, 12:40 AM JeongWoo Choi @.***> wrote:

@scmurdock https://github.com/scmurdock Any idea ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/udacity/nd029-c2-apache-spark-and-spark-streaming-starter/issues/1#issuecomment-983371642, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJBTBX4TREMQ6VR4LIZO6LUOXGQBANCNFSM5JDCL7VA .

rightx2 commented 2 years ago

@scmurdock

When I run the command kafka-console-consumer.sh --topic "redis-server" --bootstrap-server localhost:9092 --from-beginning in docker-compose exec kafka /bin/bash environment, it prints out the data clearly:

...
{"key":"UmVzZXJ2YXRpb24=","existType":"NONE","ch":false,"incr":false,"zSetEntries":[{"element":"eyJyZXNlcnZhdGlvbklkIjoiMTYzODM2NDE5MDMxOCIsImN1c3RvbWVySWQiOiIzNzcyNDQ1MjMiLCJjdXN0b21lck5hbWUiOiJCZW4gQWJyYW0iLCJ0cnVja051bWJlciI6IjU4NDAiLCJyZXNlcnZhdGlvbkRhdGUiOiIyMDIxLTEyLTAxVDEzOjA5OjUwLjMxOFoiLCJjaGVja0luU3RhdHVzIjoiQ2hlY2tlZE91dCIsIm9yaWdpbiI6IlRleGFzIiwiZGVzdGluYXRpb24iOiJUZW5uZXNzZWUifQ==","score":0.0}],"zsetEntries":[{"element":"eyJyZXNlcnZhdGlvbklkIjoiMTYzODM2NDE5MDMxOCIsImN1c3RvbWVySWQiOiIzNzcyNDQ1MjMiLCJjdXN0b21lck5hbWUiOiJCZW4gQWJyYW0iLCJ0cnVja051bWJlciI6IjU4NDAiLCJyZXNlcnZhdGlvbkRhdGUiOiIyMDIxLTEyLTAxVDEzOjA5OjUwLjMxOFoiLCJjaGVja0luU3RhdHVzIjoiQ2hlY2tlZE91dCIsIm9yaWdpbiI6IlRleGFzIiwiZGVzdGluYXRpb24iOiJUZW5uZXNzZWUifQ==","score":0.0}]}
{"key":"UGF5bWVudA==","existType":"NONE","ch":false,"incr":false,"zSetEntries":[{"element":"eyJjdXN0b21lcklkIjoiOTkxNDExMzIyIiwiY3VzdG9tZXJOYW1lIjoiSm9obiBBYnJhbSIsInJlc2VydmF0aW9uSWQiOiIxNjM4MzY0MTUyMzA0IiwiYW1vdW50Ijo2OTkuMX0=","score":0.0}],"zsetEntries":[{"element":"eyJjdXN0b21lcklkIjoiOTkxNDExMzIyIiwiY3VzdG9tZXJOYW1lIjoiSm9obiBBYnJhbSIsInJlc2VydmF0aW9uSWQiOiIxNjM4MzY0MTUyMzA0IiwiYW1vdW50Ijo2OTkuMX0=","score":0.0}]}
...
scmurdock commented 2 years ago

Ok, sounds good. It looks like kafka, connect, and redis are working properly.

On Wed, Dec 1, 2021 at 6:18 AM JeongWoo Choi @.***> wrote:

@scmurdock https://github.com/scmurdock

When I run the command kafka-console-consumer.sh --topic "redis-server" --bootstrap-server localhost:9092 --from-beginning in docker-compose exec kafka /bin/bash environment, it prints out the data clearly:

... {"key":"UmVzZXJ2YXRpb24=","existType":"NONE","ch":false,"incr":false,"zSetEntries":[{"element":"eyJyZXNlcnZhdGlvbklkIjoiMTYzODM2NDE5MDMxOCIsImN1c3RvbWVySWQiOiIzNzcyNDQ1MjMiLCJjdXN0b21lck5hbWUiOiJCZW4gQWJyYW0iLCJ0cnVja051bWJlciI6IjU4NDAiLCJyZXNlcnZhdGlvbkRhdGUiOiIyMDIxLTEyLTAxVDEzOjA5OjUwLjMxOFoiLCJjaGVja0luU3RhdHVzIjoiQ2hlY2tlZE91dCIsIm9yaWdpbiI6IlRleGFzIiwiZGVzdGluYXRpb24iOiJUZW5uZXNzZWUifQ==","score":0.0}],"zsetEntries":[{"element":"eyJyZXNlcnZhdGlvbklkIjoiMTYzODM2NDE5MDMxOCIsImN1c3RvbWVySWQiOiIzNzcyNDQ1MjMiLCJjdXN0b21lck5hbWUiOiJCZW4gQWJyYW0iLCJ0cnVja051bWJlciI6IjU4NDAiLCJyZXNlcnZhdGlvbkRhdGUiOiIyMDIxLTEyLTAxVDEzOjA5OjUwLjMxOFoiLCJjaGVja0luU3RhdHVzIjoiQ2hlY2tlZE91dCIsIm9yaWdpbiI6IlRleGFzIiwiZGVzdGluYXRpb24iOiJUZW5uZXNzZWUifQ==","score":0.0}]} {"key":"UGF5bWVudA==","existType":"NONE","ch":false,"incr":false,"zSetEntries":[{"element":"eyJjdXN0b21lcklkIjoiOTkxNDExMzIyIiwiY3VzdG9tZXJOYW1lIjoiSm9obiBBYnJhbSIsInJlc2VydmF0aW9uSWQiOiIxNjM4MzY0MTUyMzA0IiwiYW1vdW50Ijo2OTkuMX0=","score":0.0}],"zsetEntries":[{"element":"eyJjdXN0b21lcklkIjoiOTkxNDExMzIyIiwiY3VzdG9tZXJOYW1lIjoiSm9obiBBYnJhbSIsInJlc2VydmF0aW9uSWQiOiIxNjM4MzY0MTUyMzA0IiwiYW1vdW50Ijo2OTkuMX0=","score":0.0}]} ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/udacity/nd029-c2-apache-spark-and-spark-streaming-starter/issues/1#issuecomment-983633068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJBTBRG56BC4I7UM36Y3PTUOYODRANCNFSM5JDCL7VA .

-- mobile 801-719-0908

rightx2 commented 2 years ago

I don't think the connect works properly because after a few hours of waiting(after start), I got some new errors from the connect:

[2021-12-01 21:03:01,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:03:01,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:04:01,224] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:04:01,224] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:04:06,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:04:06,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:05:06,224] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:05:06,225] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:05:11,225] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:05:11,225] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:06:11,225] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:06:11,226] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:06:16,226] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:06:16,226] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:07:16,226] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:07:16,227] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:07:21,227] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:07:21,227] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:08:21,227] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:08:21,228] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:08:26,228] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:08:26,228] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:09:26,228] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:09:26,229] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:09:31,229] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:09:31,229] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:10:31,229] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:10:31,230] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:10:36,230] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:10:36,230] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:11:36,230] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:11:36,231] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:11:41,231] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:11:41,231] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:12:41,232] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:12:41,232] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:12:46,232] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:12:46,233] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:13:46,233] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:13:46,233] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:13:51,233] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:13:51,234] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:14:51,234] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:14:51,234] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:14:56,235] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:14:56,235] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:15:56,235] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:15:56,236] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:16:01,236] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:16:01,236] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:17:01,236] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:17:01,237] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:17:06,237] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:17:06,237] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:18:06,238] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:18:06,238] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:18:11,238] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:18:11,238] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:19:11,239] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:19:11,239] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:19:16,239] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:19:16,239] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:20:16,240] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:20:16,240] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:20:21,240] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:20:21,241] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:21:21,241] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:21:21,241] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:21:26,241] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:21:26,242] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:22:26,242] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:22:26,242] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:22:31,242] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:22:31,243] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:23:31,243] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:23:31,243] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:23:36,243] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:23:36,244] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:24:36,244] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:24:36,244] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:24:41,244] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:24:41,245] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:25:41,245] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:25:41,245] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:25:46,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:25:46,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:26:46,246] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:26:46,246] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:26:51,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:26:51,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:27:51,246] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:27:51,247] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:27:56,247] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:27:56,247] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:28:56,247] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:28:56,248] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:29:01,248] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:29:01,248] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:30:01,248] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:30:01,249] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:30:06,249] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:30:06,249] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:31:06,249] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:31:06,250] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:31:11,250] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:31:11,250] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:32:11,251] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:32:11,251] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:32:16,251] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:32:16,251] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:33:16,252] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:33:16,252] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:33:21,252] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:33:21,252] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:34:21,253] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:34:21,253] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:34:26,253] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:34:26,254] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:35:26,254] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:35:26,254] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:35:31,254] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:35:31,255] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:36:31,255] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:36:31,255] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:36:36,255] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:36:36,256] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:37:36,256] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:37:36,256] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:37:41,257] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:37:41,257] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:38:41,257] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:38:41,257] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:38:46,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:38:46,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:39:46,258] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:39:46,258] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:39:51,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:39:51,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:40:51,258] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:40:51,259] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:40:56,259] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:40:56,259] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:41:56,259] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:41:56,260] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:42:01,260] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:42:01,260] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:43:01,261] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:43:01,261] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:43:06,261] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:43:06,262] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:44:06,262] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:44:06,262] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:44:11,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:44:11,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:45:11,263] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:45:11,263] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:45:16,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:45:16,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:46:16,263] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:46:16,263] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:46:21,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:46:21,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:47:21,263] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:47:21,264] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:47:26,264] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:47:26,264] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:48:26,264] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:48:26,265] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:48:31,265] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:48:31,265] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:49:31,266] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:49:31,266] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:49:36,266] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:49:36,266] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:50:36,267] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:50:36,267] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:50:41,267] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:50:41,267] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:51:41,268] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:51:41,268] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:51:46,268] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:51:46,268] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:52:46,269] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:52:46,269] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:52:51,269] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:52:51,270] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2021-12-01 21:53:51,270] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2021-12-01 21:53:51,270] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
[2021-12-01 21:53:56,270] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2021-12-01 21:53:56,271] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
rightx2 commented 2 years ago 
=> docker ps
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED        STATUS                   PORTS                                                                                            NAMES
1bc70859036a   gcr.io/simulation-images/banking-simulation           "java -jar /banking.…"   21 hours ago   Up 9 hours               4567/tcp                                                                                         nd029-c2-apache-spark-and-spark-streaming-starter_banking-simulation_1
b11d3ce41aec   gcr.io/simulation-images/trucking-simulation          "java -jar /trucking…"   21 hours ago   Up 9 hours               4567/tcp                                                                                         nd029-c2-apache-spark-and-spark-streaming-starter_trucking-simulation_1
22404fe7f12a   gcr.io/simulation-images/stedi                        "java -jar /stedi.jar"   21 hours ago   Up 9 hours               0.0.0.0:4567->4567/tcp, :::4567->4567/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_stedi_1
2015b3407408   gcr.io/simulation-images/kafka-connect-redis-source   "/bin/sh -c 'connect…"   21 hours ago   Up 9 hours (unhealthy)   0.0.0.0:5005->5005/tcp, :::5005->5005/tcp, 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp   nd029-c2-apache-spark-and-spark-streaming-starter_connect_1
823bc0cd49fb   confluentinc/cp-kafka:5.5.1                           "/etc/confluent/dock…"   21 hours ago   Up 9 hours               0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_kafka_1
482c2f4c9587   bitnami/spark:3-debian-10                             "/opt/bitnami/script…"   21 hours ago   Up 9 hours                                                                                                                nd029-c2-apache-spark-and-spark-streaming-starter_spark-worker-1_1
5128d3a83c40   redis:6.0.6                                           "docker-entrypoint.s…"   21 hours ago   Up 9 hours               0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_redis_1
99ee33d79067   confluentinc/cp-zookeeper:5.5.1                       "/etc/confluent/dock…"   21 hours ago   Up 9 hours               2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp                                    nd029-c2-apache-spark-and-spark-streaming-starter_zookeeper_1
a7a75657a28d   bitnami/spark:3-debian-10                             "/opt/bitnami/script…"   21 hours ago   Up 9 hours               0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                        nd029-c2-apache-spark-and-spark-streaming-starter_spark_1
rightx2 commented 2 years ago

@scmurdock May I see your logs (docker ps, docker-compose logs connect, etc) ? I'd like to compare it with me if yours is really working well

scmurdock commented 2 years ago

The problem appears to be with the version of spark in the spark container. This same code worked previously.

On Wed, Dec 1, 2021, 2:55 PM JeongWoo Choi @.***> wrote:

I don't think the connect works properly because after some hours of waiting(after start), I got some new errors from the connect:

[2021-12-01 21:03:01,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:03:01,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:04:01,224] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:04:01,224] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:04:06,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:04:06,224] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:05:06,224] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:05:06,225] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:05:11,225] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:05:11,225] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:06:11,225] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:06:11,226] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:06:16,226] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:06:16,226] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:07:16,226] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:07:16,227] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:07:21,227] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:07:21,227] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:08:21,227] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:08:21,228] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:08:26,228] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:08:26,228] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:09:26,228] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:09:26,229] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:09:31,229] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:09:31,229] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:10:31,229] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:10:31,230] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:10:36,230] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:10:36,230] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:11:36,230] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:11:36,231] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:11:41,231] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:11:41,231] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:12:41,232] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:12:41,232] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:12:46,232] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:12:46,233] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:13:46,233] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:13:46,233] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:13:51,233] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:13:51,234] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:14:51,234] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:14:51,234] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:14:56,235] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:14:56,235] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:15:56,235] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:15:56,236] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:16:01,236] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:16:01,236] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:17:01,236] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:17:01,237] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:17:06,237] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:17:06,237] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:18:06,238] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:18:06,238] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:18:11,238] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:18:11,238] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:19:11,239] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:19:11,239] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:19:16,239] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:19:16,239] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:20:16,240] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:20:16,240] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:20:21,240] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:20:21,241] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:21:21,241] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:21:21,241] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:21:26,241] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:21:26,242] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:22:26,242] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:22:26,242] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:22:31,242] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:22:31,243] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:23:31,243] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:23:31,243] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:23:36,243] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:23:36,244] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:24:36,244] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:24:36,244] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:24:41,244] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:24:41,245] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:25:41,245] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:25:41,245] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:25:46,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:25:46,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:26:46,246] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:26:46,246] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:26:51,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:26:51,246] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:27:51,246] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:27:51,247] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:27:56,247] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:27:56,247] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:28:56,247] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:28:56,248] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:29:01,248] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:29:01,248] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:30:01,248] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:30:01,249] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:30:06,249] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:30:06,249] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:31:06,249] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:31:06,250] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:31:11,250] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:31:11,250] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:32:11,251] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:32:11,251] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:32:16,251] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:32:16,251] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:33:16,252] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:33:16,252] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:33:21,252] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:33:21,252] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:34:21,253] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:34:21,253] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:34:26,253] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:34:26,254] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:35:26,254] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:35:26,254] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:35:31,254] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:35:31,255] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:36:31,255] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:36:31,255] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:36:36,255] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:36:36,256] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:37:36,256] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:37:36,256] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:37:41,257] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:37:41,257] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:38:41,257] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:38:41,257] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:38:46,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:38:46,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:39:46,258] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:39:46,258] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:39:51,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:39:51,258] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:40:51,258] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:40:51,259] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:40:56,259] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:40:56,259] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:41:56,259] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:41:56,260] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:42:01,260] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:42:01,260] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:43:01,261] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:43:01,261] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:43:06,261] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:43:06,262] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:44:06,262] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:44:06,262] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:44:11,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:44:11,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:45:11,263] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:45:11,263] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:45:16,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:45:16,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:46:16,263] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:46:16,263] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:46:21,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:46:21,263] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:47:21,263] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:47:21,264] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:47:26,264] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:47:26,264] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:48:26,264] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:48:26,265] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:48:31,265] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:48:31,265] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:49:31,266] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:49:31,266] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:49:36,266] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:49:36,266] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:50:36,267] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:50:36,267] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:50:41,267] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:50:41,267] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:51:41,268] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:51:41,268] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:51:46,268] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:51:46,268] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:52:46,269] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:52:46,269] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:52:51,269] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:52:51,270] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116) [2021-12-01 21:53:51,270] INFO WorkerSourceTask{id=redis-config-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426) [2021-12-01 21:53:51,270] INFO WorkerSourceTask{id=redis-config-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443) [2021-12-01 21:53:56,270] ERROR WorkerSourceTask{id=redis-config-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448) [2021-12-01 21:53:56,271] ERROR WorkerSourceTask{id=redis-config-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/udacity/nd029-c2-apache-spark-and-spark-streaming-starter/issues/1#issuecomment-984088420, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJBTBVSBHMFFVL4VBEO2YLUO2KVBANCNFSM5JDCL7VA .

rightx2 commented 2 years ago

@scmurdock And what would that the "version of spark" be?

rightx2 commented 2 years ago

@scmurdock I've solved issue of Failed to find data source: kafka. So below code works without any error:

kafkaRawStreamingDF = (
    spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "balance-updates")
    .option("startingOffsets", "earliest")
    .load()
)

However, when I tried to run below code after that:

kafkaStreamingDF = kafkaRawStreamingDF.selectExpr(
    "cast(key as string) key", "cast(value as string) value"
)
kafkaStreamingDF.writeStream.outputMode("append").format(
    "console"
).start().awaitTermination()

This caused errors:

21/12/07 13:11:42 INFO Executor: Starting executor ID driver on host 2726a0665222
21/12/07 13:11:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42251.
21/12/07 13:11:42 INFO NettyBlockTransferService: Server created on 2726a0665222:42251
21/12/07 13:11:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/12/07 13:11:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 2726a0665222, 42251, None)
21/12/07 13:11:42 INFO BlockManagerMasterEndpoint: Registering block manager 2726a0665222:42251 with 408.9 MiB RAM, BlockManagerId(driver, 2726a0665222, 42251, None)
21/12/07 13:11:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 2726a0665222, 42251, None)
21/12/07 13:11:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 2726a0665222, 42251, None)
21/12/07 13:11:43 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/bitnami/spark/spark-warehouse').
21/12/07 13:11:43 INFO SharedState: Warehouse path is 'file:/opt/bitnami/spark/spark-warehouse'.
21/12/07 13:11:44 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/07 13:11:44 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/07 13:11:45 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
21/12/07 13:11:46 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/07 13:11:46 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
21/12/07 13:11:47 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0-1, groupId=spark-kafka-source-4bb4510f-5a3e-43bb-b557-0b054d833df5-670584047-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
...

I think this one is truly related with connectservice.

Here is the log of docker-compose logs connect:

connect_1              | [2021-12-07 12:54:41,766] ERROR Error converting event to JSON (org.apache.kafka.connect.redis.RedisSourceTask:102)
connect_1              | com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.moilioncircle.redis.replicator.event.PreRdbSyncEvent and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
connect_1              |    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
connect_1              |    at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191)
connect_1              |    at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:404)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:71)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:33)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:4094)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3404)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.getSourceRecord(RedisSourceTask.java:99)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.poll(RedisSourceTask.java:77)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:272)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:239)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
connect_1              |    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
connect_1              |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
connect_1              |    at java.lang.Thread.run(Thread.java:748)
connect_1              | [2021-12-07 12:54:41,771] INFO heartbeat started. (com.moilioncircle.redis.replicator.RedisSocketReplicator:239)
connect_1              | [2021-12-07 12:54:41,818] ERROR Error converting event to JSON (org.apache.kafka.connect.redis.RedisSourceTask:102)
connect_1              | com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.moilioncircle.redis.replicator.event.PreCommandSyncEvent and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
connect_1              |    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
connect_1              |    at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191)
connect_1              |    at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:404)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:71)
connect_1              |    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:33)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
connect_1              |    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:4094)
connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3404)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.getSourceRecord(RedisSourceTask.java:99)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.poll(RedisSourceTask.java:77)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:272)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:239)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
connect_1              |    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
connect_1              |    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
connect_1              |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
connect_1              |    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
connect_1              |    at java.lang.Thread.run(Thread.java:748)
OneCricketeer commented 2 years ago

The error is at these lines

connect_1              |    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3404)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.getSourceRecord(RedisSourceTask.java:99)
connect_1              |    at org.apache.kafka.connect.redis.RedisSourceTask.poll(RedisSourceTask.java:77)

Report the issue in this repo

https://github.com/Aegeaner/kafka-connector-redis/blob/master/src/main/java/org/apache/kafka/connect/redis/RedisSourceTask.java#L99

OneCricketeer commented 2 years ago

Also, regarding the spark code, unless you are trying to run PySpark outside of Docker, you want this when doing spark-submit inside of the spark master/worker

- .option("kafka.bootstrap.servers", "localhost:9092")
+.option("kafka.bootstrap.servers", "kafka:19092")   # this is the bootstrap server used by connect
rightx2 commented 2 years ago

@OneCricketeer Thanks for the code correction. After changing the code from localhost to kafka, 1 error has been removed. When using localhost:9092 in the code:

21/12/08 06:03:25 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-b711aab8-9dbf-4c1f-a748-8bc71ccf27d6--1220610823-driver-0-1, groupId=spark-kafka-source-b711aab8-9dbf-4c1f-a748-8bc71ccf27d6--1220610823-driver-0] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
21/12/08 06:03:25 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-b711aab8-9dbf-4c1f-a748-8bc71ccf27d6--1220610823-driver-0-1, groupId=spark-kafka-source-b711aab8-9dbf-4c1f-a748-8bc71ccf27d6--1220610823-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

When using kafka:9092 in the code:

21/12/08 06:02:40 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-1eef1a62-8901-4a46-b993-d906713bb95b--1220610823-driver-0-1, groupId=spark-kafka-source-1eef1a62-8901-4a46-b993-d906713bb95b--1220610823-driver-0] Connection to node 0 (/127.0.0.1:9092) could not be established. Broker may not be available.
rightx2 commented 2 years ago

@OneCricketeer And about RedisSourceTask problem, do you think it could have something to do with Redis version as well?

OneCricketeer commented 2 years ago

kafka:19092, not kafka:9092

I'm not sure, I've never used Redis, but it's possible seeing that the readme of that project gives different instructions for Redis 5 and 6

rightx2 commented 2 years ago

@OneCricketeer After changing to kafka:19092. It perfectly worked! Thank you so much. You saved my days!

rightx2 commented 2 years ago

I think we have to make a PR. @OneCricketeer Do you want to contribute? or You want me to do that?

OneCricketeer commented 2 years ago

Like I said, I don't use Redis, so I'm not sure all of the classes that would need fixed in the Jackson ObjectMapper

rightx2 commented 2 years ago

@OneCricketeer I will experiment with some other codes and see all work well, then