search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.82k stars 1.29k forks source link

Kafka-Connect Readiness and Liveness failed error #3784

Closed prash911 closed 4 years ago

prash911 commented 4 years ago

Hello Everyone, I am trying to deploy kafka-connect on strimzi operator but getting Readiness and Liveness failed error.looked for this in issues,but did not found any working solution.

Strimzi operator version : 0.19.0

Kafka version : 2.5.0

strimzi/kafka-connect:0.11.4-kafka-2.1.0

Kafka is also running in the same namespace with following bootstrap data-kafka-bootstrap ClusterIP 172.20.203.132 <none> 9091/TCP,9092/TCP,9093/TCP,9404/TCP 24d

Following is my kafka-connect config.I have not configured any readiness or liveness in this config. With Readiness and Liveness probe Disabled

versionNumber: 2.5.0
replicaCount: 1
namespace: kafka-cluster
bootstrapServersName: "data-kafka-bootstrap:9092"

image:
  repository: strimzi/kafka-connect
  tag: 0.11.4-kafka-2.1.0
  pullPolicy: Always

tls:
  certname : data-cluster-ca-cert

imagePullSecrets: []
nameOverride: "data"
fullnameOverride: ""

config_storage_replication_factor: 3
offset_storage_replication_factor: 3
status_storage_replication_factor: 3

Here is the describe output

4s         Normal    Scheduled           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Successfully assigned kafka-cluster/data-kafka-connect-cluster-connect-74d98646d8-d825l to ip-10-0-1-15.us-east-2.compute.internal
13s         Normal    Pulling             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Pulling image "strimzi/kafka-connect:0.11.4-kafka-2.1.0"
12s         Normal    Pulled              pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Successfully pulled image "strimzi/kafka-connect:0.11.4-kafka-2.1.0"
12s         Normal    Created             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Created container data-kafka-connect-cluster-connect
11s         Normal    Started             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Started container data-kafka-connect-cluster-connect
14s         Normal    SuccessfulCreate    replicaset/data-kafka-connect-cluster-connect-74d98646d8   Created pod: data-kafka-connect-cluster-connect-74d98646d8-d825l
14s         Normal    NoPods              poddisruptionbudget/data-kafka-connect-cluster-connect     No matching pods found
14s         Normal    ScalingReplicaSet   deployment/data-kafka-connect-cluster-connect              Scaled up replica set data-kafka-connect-cluster-connect-74d98646d8 to 1
6m59s       Warning   BackOff             pod/kafka-connect-cluster-connect-596b456dbf-5s2qq         Back-off restarting failed container
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Readiness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Readiness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused

Although strimzi Operator has by default this configured.

  livenessProbe:
    initialDelaySeconds: 10
    periodSeconds: 30
  readinessProbe:
    initialDelaySeconds: 10
    periodSeconds: 30

I have also tried configuring kafka-connect with Readiness and Liveness probe with following config.But still same result.

With Readiness and Liveness probe Enabled

    livenessProbe:
      httpGet:
          path: /connectors
          port: 8083
      initialDelaySeconds: 60
      periodSeconds: 5
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 10
    readinessProbe:
      httpGet:
          path: /connectors
          port: 8083
      initialDelaySeconds: 40
      periodSeconds: 10
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 3

Please let me know if anything else required.

Thanks in advance.

scholzj commented 4 years ago

For the probes, you can only configure the timings and thresholds. You cannot configure what exactly does the probe do. That is not configurable. Could you share the YAML of the KafkaConnect custom resource and the logs from the Kafka Connect pod(s)?

scholzj commented 4 years ago

Also, if you are running Strimzi 0.11.4, it for sure doesn't support Kafka 2.5.0.

prash911 commented 4 years ago

Strimzi operator version is 0.19.0 .This one strimzi/kafka-connect:0.11.4-kafka-2.1.0 is latest kafka connect image available.Is that compatible?

The probes config, i referred from here https://github.com/confluentinc/cp-helm-charts/blob/3434a1e1ae4992a4c04b528f36408958aab6b07a/charts/cp-kafka-connect/values.yaml#L125

Following is the kafka-connect pod log

Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=data-kafka-bootstrap:9092
# REST Listeners
rest.port=8083
rest.advertised.host.name=10.0.1.51
rest.advertised.port=8083
# Plugins
plugin.path=/opt/kafka/plugins
# Provided configuration
offset.storage.topic=connect-cluster-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect-cluster-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=connect-cluster
status.storage.topic=connect-cluster-status
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

security.protocol=SSL
producer.security.protocol=SSL
consumer.security.protocol=SSL
# TLS / SSL
ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
ssl.truststore.password=-mM7oU_XEQ01PIMzaPfTP1aJJCGeMD4B
ssl.truststore.type=PKCS12

producer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
producer.ssl.truststore.password=-mM7oU_XEQ01PIMzaPfTP1aJJCGeMD4B

consumer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
consumer.ssl.truststore.password=-mM7oU_XEQ01PIMzaPfTP1aJJCGeMD4B

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
2020-10-12 08:52:06,225 INFO Kafka Connect distributed worker initializing ... (org.apache.kafka.connect.cli.ConnectDistributed) [main]
2020-10-12 08:52:06,237 INFO WorkerInfo values:
    jvm.args = -Xms128M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/kafka, -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
    jvm.spec = Oracle Corporation, OpenJDK 64-Bit Server VM, 1.8.0_212, 25.212-b04
    jvm.classpath = /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.5.jar:/opt/kafka/bin/../libs/compileScala.mapping:/opt/kafka/bin/../libs/compileScala.mapping.asc:/opt/kafka/bin/../libs/connect-api-2.1.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.1.0.jar:/opt/kafka/bin/../libs/connect-file-2.1.0.jar:/opt/kafka/bin/../libs/connect-json-2.1.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.1.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.1.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.7.jar:/opt/kafka/bin/../libs/jackson-core-2.9.7.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.7.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.7.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.7.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.7.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.27.jar:/opt/kafka/bin/../libs/jersey-common-2.27.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/bin/../libs/jersey-server-2.27.jar:/opt/kafka/bin/../libs/jetty-client-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-http-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-io-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-security-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-server-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jetty-util-9.4.12.v20180830.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka-clients-2.1.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.12-2.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.1.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.1.0.jar:/opt/kafka/bin/../libs/kafka_2.12-2.1.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.12-2.1.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.5.4.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.14.2.jar:/opt/kafka/bin/../libs/scala-library-2.12.7.jar:/opt/kafka/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.12.7.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.10.jar:/opt/kafka/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/bin/../libs/zstd-jni-1.3.5-4.jar
    os.spec = Linux, amd64, 4.14.186-146.268.amzn2.x86_64
    os.vcpus = 1
 (org.apache.kafka.connect.runtime.WorkerInfo) [main]
2020-10-12 08:52:06,243 INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed) [main]
2020-10-12 08:52:09,107 INFO Registered loader: sun.misc.Launcher$AppClassLoader@764c12b6 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,107 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,107 INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,108 INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,110 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,110 INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,110 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,110 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,110 INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,110 INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,111 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,112 INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,112 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,112 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,112 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,112 INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,112 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,113 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,114 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,114 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,114 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,114 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,114 INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,114 INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,115 INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,119 INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,119 INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,119 INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,120 INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,121 INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,123 INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,123 INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,125 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,125 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,125 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,125 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,125 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,126 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,126 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,126 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,126 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,128 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,129 INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,129 INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,129 INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,129 INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-12 08:52:09,162 INFO DistributedConfig values:
    access.control.allow.methods =
    access.control.allow.origin =
    bootstrap.servers = [data-kafka-bootstrap:9092]
    client.dns.lookup = default
    client.id =
    config.providers = []
    config.storage.replication.factor = 3
    config.storage.topic = connect-cluster-configs
    connections.max.idle.ms = 540000
    group.id = connect-cluster
    header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
    heartbeat.interval.ms = 3000
    internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
    internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
    key.converter = class org.apache.kafka.connect.json.JsonConverter
    listeners = null
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    offset.flush.interval.ms = 60000
    offset.flush.timeout.ms = 5000
    offset.storage.partitions = 25
    offset.storage.replication.factor = 3
    offset.storage.topic = connect-cluster-offsets
    plugin.path = [/opt/kafka/plugins]
    rebalance.timeout.ms = 60000
    receive.buffer.bytes = 32768
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 40000
    rest.advertised.host.name = 10.0.1.51
    rest.advertised.listener = null
    rest.advertised.port = 8083
    rest.extension.classes = []
    rest.host.name = null
    rest.port = 8083
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.client.auth = none
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
    ssl.truststore.password = [hidden]
    ssl.truststore.type = PKCS12
    status.storage.partitions = 5
    status.storage.replication.factor = 3
    status.storage.topic = connect-cluster-status
    task.shutdown.graceful.timeout.ms = 5000
    value.converter = class org.apache.kafka.connect.json.JsonConverter
    worker.sync.timeout.ms = 3000
    worker.unsync.backoff.ms = 300000
 (org.apache.kafka.connect.runtime.distributed.DistributedConfig) [main]
2020-10-12 08:52:09,163 INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils) [main]
2020-10-12 08:52:09,179 INFO AdminClientConfig values:
    bootstrap.servers = [data-kafka-bootstrap:9092]
    client.dns.lookup = default
    client.id =
    connections.max.idle.ms = 300000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 120000
    retries = 5
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
    ssl.truststore.password = [hidden]
    ssl.truststore.type = PKCS12
 (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,643 WARN The configuration 'producer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,643 WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,643 WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,643 WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'consumer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'producer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'consumer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-12 08:52:09,644 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfi
scholzj commented 4 years ago

If you use Strimzi 0.19.0, you cannot use the image strimzi/kafka-connect:0.11.4-kafka-2.1.0. You have to use the images from 0.19.0. For example strimzi/kafka-connect:0.19.0-kafka-2.5.0.

prash911 commented 4 years ago

So strimzi/kafka-connect:0.19.0-kafka-2.5.0 is the latest kafka-connect image available. Do i need to downgrade the kafka version to use kafka-connect?

scholzj commented 4 years ago

Sorry, not sure what exactly do you mean with Do i need to downgrade the kafka version to use kafka-connect? Strimzi 0.19.0 supports only Kafka 2.4.0, 2.4.1 and 2.5.0. If you want to use Kafka Connect 2.1.0, you would need to use older version of Strimzi which supports Kafka 2.1.0. But in general, newer connect should work with older brokers so maybe you do not need to do so.

prash911 commented 4 years ago

I see this version only last updated. https://hub.docker.com/r/strimzi/kafka-connect/tags

Is there going to be a new update going to come for kafka-2.5.0?

scholzj commented 4 years ago

Oh, right ... you are right of course. The kafka-connect image is not used anymore. Sorry, I completely forgot about it since it is so long ago. Connect is inside the kafka image. So here: https://hub.docker.com/r/strimzi/kafka/tags

prash911 commented 4 years ago

So can i use this image just for Kafka Connect? I am using following kafka image strimzi/kafka:0.19.0-kafka-2.5.0

scholzj commented 4 years ago

Yeah, you can use it in the Kafka Connect deployment with Strimzi.

prash911 commented 4 years ago

Thanks for your help. Is there a documentation on how can i use this kafka-connect? Even by using new image strimzi/kafka:0.19.0-kafka-2.5.0 still getting same error.

scholzj commented 4 years ago

Well, the log you pasted before does not show any errors. So you should check the new log for the new errors. I also don't think you share the KafkaConnect custom resource YAML yet - that would be useful as well.

prash911 commented 4 years ago

For the KafkaConnect custom resource YAMLdo you mean the kafka Connect CRD from the operator?,if not please guide me which file it is? (Pls correct me if i am wrong somewhere or missed something)

Kafka and Kafka connect are running under same namespace.

Below is the helm template

# Source: kafka-connect/templates/connect-deployment.yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
    name: data-kafka-cluster
    namespace: kafka-cluster
    labels:
        app: data-kafka-cluster
        chart: kafka-connect-0.1.0
        release: kafka-connect-new
        heritage: Helm
spec:
    version: 2.5.0
    replicas: 1
    bootstrapServers: data-kafka-bootstrap:9092
    image: "strimzi/kafka:0.19.0-kafka-2.5.0"
    imagePullPolicy: Always
    tls:
        trustedCertificates:
            - secretName: data-cluster-ca-cert
              certificate: ca.crt
    env:
      livenessProbe:
        httpGet:
            path: /
            port: 8083
        initialDelaySeconds: 90
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 10
      readinessProbe:
        httpGet:
            path: /
            port: 8083
        initialDelaySeconds: 120
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 3

    config:
        group.id: connect-cluster
        offset.storage.topic: connect-cluster-offsets
        config.storage.topic: connect-cluster-configs
        status.storage.topic: connect-cluster-status
        config.storage.replication.factor: 3
        offset.storage.replication.factor: 3
        status.storage.replication.factor: 3

Following is the new log of the pod 

Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=data-kafka-bootstrap:9092
# REST Listeners
rest.port=8083
rest.advertised.host.name=10.0.1.51 
rest.advertised.port=8083
# Plugins
plugin.path=/opt/kafka/plugins
# Provided configuration
offset.storage.topic=connect-cluster-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect-cluster-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=connect-cluster
status.storage.topic=connect-cluster-status
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

security.protocol=SSL
producer.security.protocol=SSL
consumer.security.protocol=SSL
admin.security.protocol=SSL
# TLS / SSL
ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
ssl.truststore.password=[hidden]
ssl.truststore.type=PKCS12

producer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
producer.ssl.truststore.password=[hidden]

consumer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
consumer.ssl.truststore.password=[hidden]

admin.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
admin.ssl.truststore.password=[hidden]

2020-10-14 01:53:05,273 INFO WorkerInfo values: 
    jvm.args = -Xms128M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/kafka, -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
    jvm.spec = Oracle Corporation, OpenJDK 64-Bit Server VM, 11.0.7, 11.0.7+10-LTS
    jvm.classpath = /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/bcpkix-jdk15on-1.62.jar:/opt/kafka/bin/../libs/bcprov-jdk15on-1.60.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.5.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.5.0.jar:/opt/kafka/bin/../libs/connect-file-2.5.0.jar:/opt/kafka/bin/../libs/connect-json-2.5.0.jar:/opt/kafka/bin/../libs/connect-mirror-2.5.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-2.5.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.5.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.5.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.0.108.jar:/opt/kafka/bin/../libs/gson-2.8.6.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/bin/../libs/jackson-annotations-2.10.2.jar:/opt/kafka/bin/../libs/jackson-core-2.10.2.jar:/opt/kafka/bin/../libs/jackson-databind-2.10.2.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.10.2.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.10.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.10.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.12-2.10.2.jar:/opt/kafka/bin/../libs/jaeger-client-1.1.0.jar:/opt/kafka/bin/../libs/jaeger-core-1.1.0.jar:/opt/kafka/bin/../libs/jaeger-thrift-1.1.0.jar:/opt/kafka/bin/../libs/jaeger-tracerresolver-1.1.0.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javassist-3.26.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.28.jar:/opt/kafka/bin/../libs/jersey-common-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/bin/../libs/jersey-server-2.28.jar:/opt/kafka/bin/../libs/jetty-client-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-http-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-io-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-security-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-server-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-util-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.12.0.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/json-smart-1.1.1.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka-agent.jar:/opt/kafka/bin/../libs/kafka-clients-2.5.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.12-2.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.5.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.5.0.jar:/opt/kafka/bin/../libs/kafka_2.12-2.5.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.12-2.5.0.jar:/opt/kafka/bin/../libs/keycloak-common-10.0.0.jar:/opt/kafka/bin/../libs/keycloak-core-10.0.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.3.50.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.3.50.jar:/opt/kafka/bin/../libs/libthrift-0.13.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/opt/kafka/bin/../libs/okhttp-4.2.2.jar:/opt/kafka/bin/../libs/okio-2.2.2.jar:/opt/kafka/bin/../libs/opa-authorizer-0.4.1.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-kafka-client-0.1.12.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../libs/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.12-2.1.3.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/opt/kafka/bin/../libs/scala-library-2.12.10.jar:/opt/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.12.10.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/tracing-agent.jar:/opt/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/bin/../libs/zookeeper-3.5.7.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.4-7.jar
    os.spec = Linux, amd64, 4.14.186-146.268.amzn2.x86_64
    os.vcpus = 1
 (org.apache.kafka.connect.runtime.WorkerInfo) [main]
2020-10-14 01:53:05,280 INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed) [main]
2020-10-14 01:53:11,101 INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@3d4eac69 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,101 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,101 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,101 INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,102 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,103 INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,103 INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,103 INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,104 INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,104 INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,104 INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,104 INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,104 INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,104 INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,105 INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,105 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,106 INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,107 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,108 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,108 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,108 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,108 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,108 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,109 INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,109 INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,109 INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,109 INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,109 INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,113 INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,114 INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,114 INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,115 INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,115 INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,116 INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,117 INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,118 INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,118 INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,118 INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,118 INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,119 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,120 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,121 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,121 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,122 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,122 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,123 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,123 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,123 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,124 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,124 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,124 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,124 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,124 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,127 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,128 INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,128 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,128 INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,128 INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,128 INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,128 INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,129 INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,129 INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,129 INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-14 01:53:11,220 INFO DistributedConfig values: 
    access.control.allow.methods = 
    access.control.allow.origin = 
    admin.listeners = null
    bootstrap.servers = [data-kafka-bootstrap:9092]
    client.dns.lookup = default
    client.id = 
    config.providers = []
    config.storage.replication.factor = 3
    config.storage.topic = connect-cluster-configs
    connect.protocol = sessioned
    connections.max.idle.ms = 540000
    connector.client.config.override.policy = None
    group.id = connect-cluster
    header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
    heartbeat.interval.ms = 3000
    inter.worker.key.generation.algorithm = HmacSHA256
    inter.worker.key.size = null
    inter.worker.key.ttl.ms = 3600000
    inter.worker.signature.algorithm = HmacSHA256
    inter.worker.verification.algorithms = [HmacSHA256]
    internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
    internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
    key.converter = class org.apache.kafka.connect.json.JsonConverter
    listeners = null
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    offset.flush.interval.ms = 60000
    offset.flush.timeout.ms = 5000
    offset.storage.partitions = 25
    offset.storage.replication.factor = 3
    offset.storage.topic = connect-cluster-offsets
    plugin.path = [/opt/kafka/plugins]
    rebalance.timeout.ms = 60000
    receive.buffer.bytes = 32768
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 40000
    rest.advertised.host.name = 10.0.1.51
    rest.advertised.listener = null
    rest.advertised.port = 8083
    rest.extension.classes = []
    rest.host.name = null
    rest.port = 8083
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    scheduled.rebalance.max.delay.ms = 300000
    security.protocol = SSL
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.client.auth = none
    ssl.enabled.protocols = [TLSv1.2]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.2
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
    ssl.truststore.password = [hidden]
    ssl.truststore.type = PKCS12
    status.storage.partitions = 5
    status.storage.replication.factor = 3
    status.storage.topic = connect-cluster-status
    task.shutdown.graceful.timeout.ms = 5000
    topic.tracking.allow.reset = true
    topic.tracking.enable = true
    value.converter = class org.apache.kafka.connect.json.JsonConverter
    worker.sync.timeout.ms = 3000
    worker.unsync.backoff.ms = 300000
 (org.apache.kafka.connect.runtime.distributed.DistributedConfig) [main]
2020-10-14 01:53:11,224 INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils) [main]
2020-10-14 01:53:11,248 INFO AdminClientConfig values: 
    bootstrap.servers = [data-kafka-bootstrap:9092]
    client.dns.lookup = default
    client.id = 
    connections.max.idle.ms = 300000
    default.api.timeout.ms = 60000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    security.providers = null
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.2
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
    ssl.truststore.password = [hidden]
    ssl.truststore.type = PKCS12
 (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,740 WARN The configuration 'producer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'admin.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'consumer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'producer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,741 WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,742 WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,742 WARN The configuration 'consumer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,742 WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,742 WARN The configuration 'ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,742 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,742 WARN The configuration 'admin.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'consumer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'producer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'ssl.truststore.type' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,743 WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,744 WARN The configuration 'ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,744 WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,744 WARN The configuration 'admin.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-14 01:53:11,744 INFO Kafka version: 2.5.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2020-10-14 01:53:11,745 INFO Kafka commitId: 66563e712b0b9f84 (org.apache.kafka.common.utils.AppInfoParser) [main]
2020-10-14 01:53:11,745 INFO Kafka startTimeMs: 1602640391744 (org.apache.kafka.common.utils.AppInfoParser) [main]
2020-10-14 01:53:12,098 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (data-kafka-bootstrap/172.20.203.132:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2020-10-14 01:53:12,209 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (data-kafka-bootstrap/172.20.203.132:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2020-10-14 01:53:41,751 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1602640421750) timed out at 1602640421751 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
2020-10-14 01:53:42,461 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (data-kafka-bootstrap/172.20.203.132:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.  (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2020-10-14 01:54:10,571 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (data-kafka-bootstrap/172.20.203.132:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2020-10-14 01:54:11,578 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (data-kafka-bootstrap/172.20.203.132:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2020-10-14 01:54:11,751 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1602640451751) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
2020-10-14 01:54:11,755 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
    at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
    at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
    at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602640451749) timed out at 1602640451750 after 1 attempt(s)
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
    ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602640451749) timed out at 1602640451750 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

Following is the Services details

NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                   PORT(S)                               AGE
data-kafka-0                         LoadBalancer   172.20.143.208   somelb.elb.amazonaws.com   9094:32469/TCP                        26d
data-kafka-1                         LoadBalancer   172.20.70.81     somelb.elb.amazonaws.com   9094:31410/TCP                        26d
data-kafka-2                         LoadBalancer   172.20.203.71    somelb.elb.amazonaws.com   9094:30088/TCP                        26d
data-kafka-bootstrap                 ClusterIP      172.20.203.132   <none>                     9091/TCP,9092/TCP,9093/TCP,9404/TCP   26d
data-kafka-brokers                   ClusterIP      None             <none>                     9091/TCP,9092/TCP,9093/TCP            26d
data-kafka-cluster-connect-api       ClusterIP      172.20.81.166    <none>                     8083/TCP                              42m
data-kafka-external-bootstrap        LoadBalancer   172.20.25.231    somelb.elb.amazonaws.com   9094:30162/TCP              26d
data-zookeeper-client                ClusterIP      172.20.136.153   <none>                     9404/TCP,2181/TCP                     26d
data-zookeeper-nodes                 ClusterIP      None             <none>                     2181/TCP,2888/TCP,3888/TCP            26d
scholzj commented 4 years ago

From the bootstrap address data-kafka-bootstrap:9092 I guess the Kafka cluster is using Strimzi as well? Strimzi normally has TLS on port 9093 and plaintext on port 9092. But your KafkaConnect CR connects to 9092 with TLS. That sounds like it could cause the issue you have. Are you sure your configuration is correct? Can you share also the Kafka CR you used to deploy the Kafka cluster to confirm?

prash911 commented 4 years ago

Yes, I am using the strimzi operator for kafka as well.

Following is the yaml output of kafka pod. 

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
    strimzi.io/broker-configuration-hash: 70bc9ab2fcda8a72056cf3013307c2a6a8390f474bf16b6a4a58553961893d5d117bc32ab65f59200482b06fe2befcd3d1141bb93e2ce36261e13567c67171896341c1ad64b6cd079bf8ba1eb09a522054e605186f3b8b2661365a8c69b53ffa55f673bd5acb850f4573e041fc678dd7782994cb5fbe7ee86825107b0c4ee12e
    strimzi.io/clients-ca-cert-generation: "0"
    strimzi.io/cluster-ca-cert-generation: "0"
    strimzi.io/generation: "1"
    strimzi.io/logging-hash: 21aa96756abf4220d5c5aefabfd17fd541117bdfb43d79477a9c198b4a28efb0fdff6e89b6c8dfa2823b9c9f7700a6c871e798da84a73d7b85677e5707d67852
  creationTimestamp: "2020-09-17T09:54:13Z"
  generateName: data-kafka-
  labels:
    app.kubernetes.io/instance: data
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-data
    controller-revision-hash: data-kafka-5c58c879c6
    statefulset.kubernetes.io/pod-name: data-kafka-0
    strimzi.io/cluster: data
    strimzi.io/kind: Kafka
    strimzi.io/name: data-kafka
  name: data-kafka-0
  namespace: kafka-cluster
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: data-kafka
    uid: 0563b362-1459-47a9-bce8-a2935ec950f6
  resourceVersion: "10484311"
  selfLink: /api/v1/namespaces/kafka-cluster/pods/data-kafka-0
  uid: a44d81ea-b435-4e83-a6d5-d43550ffb0f3
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: failure-domain.beta.kubernetes.io/zone
            operator: Exists
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              strimzi.io/cluster: data
              strimzi.io/name: data-kafka
          topologyKey: failure-domain.beta.kubernetes.io/zone
        weight: 100
  containers:
  - command:
    - /opt/kafka/kafka_run.sh
    env:
    - name: KAFKA_METRICS_ENABLED
      value: "true"
    - name: STRIMZI_KAFKA_GC_LOG_ENABLED
      value: "false"
    - name: KAFKA_HEAP_OPTS
      value: -Xms2000m -Xmx2000m
    image: strimzi/kafka:0.19.0-kafka-2.5.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /opt/kafka/kafka_liveness.sh
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: kafka
    ports:
    - containerPort: 9091
      name: tcp-replication
      protocol: TCP
    - containerPort: 9092
      name: tcp-clients
      protocol: TCP
    - containerPort: 9093
      name: tcp-clientstls
      protocol: TCP
    - containerPort: 9094
      name: tcp-external
      protocol: TCP
    - containerPort: 9404
      name: tcp-prometheus
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - test
        - -f
        - /var/opt/kafka/kafka-ready
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "1"
        memory: 4Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/kafka/data
      name: data
    - mountPath: /opt/kafka/cluster-ca-certs
      name: cluster-ca
    - mountPath: /opt/kafka/broker-certs
      name: broker-certs
    - mountPath: /opt/kafka/client-ca-certs
      name: client-ca-cert
    - mountPath: /opt/kafka/custom-config/
      name: kafka-metrics-and-logging
    - mountPath: /var/opt/kafka
      name: ready-files
    - mountPath: /opt/kafka/init
      name: rack-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: data-kafka-token-kxkbd
      readOnly: true
  - command:
    - /opt/stunnel/kafka_stunnel_run.sh
    env:
    - name: KAFKA_ZOOKEEPER_CONNECT
      value: data-zookeeper-client:2181
    - name: TLS_SIDECAR_LOG_LEVEL
      value: notice
    image: strimzi/kafka:0.19.0-kafka-2.5.0
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /opt/stunnel/kafka_stunnel_pre_stop.sh
    livenessProbe:
      exec:
        command:
        - /opt/stunnel/stunnel_healthcheck.sh
        - "2181"
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: tls-sidecar
    readinessProbe:
      exec:
        command:
        - /opt/stunnel/stunnel_healthcheck.sh
        - "2181"
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/tls-sidecar/kafka-brokers/
      name: broker-certs
    - mountPath: /etc/tls-sidecar/cluster-ca-certs/
      name: cluster-ca
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: data-kafka-token-kxkbd
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: data-kafka-0
  initContainers:
  - args:
    - /opt/strimzi/bin/kafka_init_run.sh
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: RACK_TOPOLOGY_KEY
      value: failure-domain.beta.kubernetes.io/zone
    image: strimzi/operator:0.19.0
    imagePullPolicy: IfNotPresent
    name: kafka-init
    resources:
      limits:
        cpu: "1"
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/kafka/init
      name: rack-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: data-kafka-token-kxkbd
      readOnly: true
  nodeName: ip-10-0-3-81.us-east-2.compute.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 0
  serviceAccount: data-kafka
  serviceAccountName: data-kafka
  subdomain: data-kafka-brokers
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-data-kafka-0
  - emptyDir: {}
    name: rack-volume
  - name: cluster-ca
    secret:
      defaultMode: 292
      secretName: data-cluster-ca-cert
  - name: broker-certs
    secret:
      defaultMode: 292
      secretName: data-kafka-brokers
  - name: client-ca-cert
    secret:
      defaultMode: 292
      secretName: data-clients-ca-cert
  - configMap:
      defaultMode: 420
      name: data-kafka-config
    name: kafka-metrics-and-logging
  - emptyDir:
      medium: Memory
    name: ready-files
  - name: data-kafka-token-kxkbd
    secret:
      defaultMode: 420
      secretName: data-kafka-token-kxkbd
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-09-17T09:54:35Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-09-17T09:54:58Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-09-17T09:54:58Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-09-17T09:54:13Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://8186557d4ff453a397299953d5e6a5090786b1d2824ecddad4349682ac543a4a
    image: strimzi/kafka:0.19.0-kafka-2.5.0
    imageID: docker-pullable://strimzi/kafka@sha256:961419d795036f5f491817a5652c8f30c531a7b3cafd245d9473812a6a632c19
    lastState: {}
    name: kafka
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2020-09-17T09:54:36Z"
  - containerID: docker://6beea6b3587c6c3cfbc10ae4b05d4bf619f83189b211c1a60cd4b56f2ae10000
    image: strimzi/kafka:0.19.0-kafka-2.5.0
    imageID: docker-pullable://strimzi/kafka@sha256:961419d795036f5f491817a5652c8f30c531a7b3cafd245d9473812a6a632c19
    lastState: {}
    name: tls-sidecar
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2020-09-17T09:54:36Z"
  hostIP: 10.0.3.81
  initContainerStatuses:
  - containerID: docker://4c930ad59d8a345668546dcf1d2623883792c200ec018f3ec73fc383a2b231b2
    image: strimzi/operator:0.19.0
    imageID: docker-pullable://strimzi/operator@sha256:d8fed9eb28c6d024670b060527681e21c6744a6c7dd2bf48450ce577382d3646
    lastState: {}
    name: kafka-init
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://4c930ad59d8a345668546dcf1d2623883792c200ec018f3ec73fc383a2b231b2
        exitCode: 0
        finishedAt: "2020-09-17T09:54:34Z"
        reason: Completed
        startedAt: "2020-09-17T09:54:31Z"
  phase: Running
  podIP: 10.0.3.178
  podIPs:
  - ip: 10.0.3.178
  qosClass: Burstable
  startTime: "2020-09-17T09:54:13Z"
scholzj commented 4 years ago

The pod does not help. You have to provide the YMAL of the Kafka custom resource ... kubectl get kafka -o yaml.

prash911 commented 4 years ago

Sorry,did not know about that.

Here it is

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta1
  kind: Kafka
  metadata:
    annotations:
      meta.helm.sh/release-name: data
      meta.helm.sh/release-namespace: kafka-cluster
    creationTimestamp: "2020-09-17T08:54:40Z"
    generation: 5
    labels:
      app.kubernetes.io/managed-by: Helm
    name: data
    namespace: kafka-cluster
    resourceVersion: "14516280"
    selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/kafka-cluster/kafkas/data
    uid: 7d9e924d-ec3c-41b4-a937-d2eb06fdb38e
  spec:
    entityOperator:
      topicOperator:
        resources:
          limits:
            cpu: 300m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
      userOperator:
        resources:
          limits:
            cpu: 300m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
    kafka:
      config:
        auto.create.topics.enable: "false"
        log.message.format.version: "2.5"
        offsets.topic.replication.factor: 3
        transaction.state.log.min.isr: 2
        transaction.state.log.replication.factor: 3
      jvmOptions:
        -Xms: 2000m
        -Xmx: 2000m
      listeners:
        external:
          tls: false
          type: loadbalancer
        plain: {}
        tls: {}
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      rack:
        topologyKey: failure-domain.beta.kubernetes.io/zone
      replicas: 3
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: "1"
          memory: 4Gi
      storage:
        class: gp2
        deleteClaim: true
        size: 100Gi
        type: persistent-claim
      version: 2.5.0
    maintenanceTimeWindows:
    - 0 0 0,4 ? * SAT *
    zookeeper:
      jvmOptions:
        -Xms: 1500m
        -Xmx: 1500m
      replicas: 3
      resources:
        limits:
          cpu: "1"
          memory: 3Gi
        requests:
          cpu: "1"
          memory: 3Gi
      storage:
        class: gp2
        deleteClaim: true
        size: 10Gi
        type: persistent-claim
  status:
    conditions:
    - lastTransitionTime: 2020-10-03T11:53:50+0000
      status: "True"
      type: Ready
    listeners:
    - addresses:
      - host: data-kafka-bootstrap.kafka-cluster.svc
        port: 9092
      bootstrapServers: data-kafka-bootstrap.kafka-cluster.svc:9092
      type: plain
    - addresses:
      - host: data-kafka-bootstrap.kafka-cluster.svc
        port: 9093
      bootstrapServers: data-kafka-bootstrap.kafka-cluster.svc:9093
      certificates:
      - |
        -----BEGIN CERTIFICATE-----
        MIIDLTCCAhWgAwIBAgIJAPl8qCNjhQatMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
        BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjAwOTE3
        MDg1NDQxWhcNMjEwOTE3MDg1NDQxWjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
        FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
        CgKCAQEA24XptyJepYTmVxXALp4gAxQ2caUTJAa95hifB7QDb6w0MaK/izXVVFnb
        BIQbJf4aOaXtJJApGvLy9VhwZoErdTgsjxybbUyGTpI78EJ6p7vd7sxNZoZ0GG5o
        oWQcR+pqy0sbivtol71IN075/TUmLugXiUcaqdXaomfelAf2VjvVXTvNNXq+qPD3
        IoX8QK9STUD1h8vQHF8DDWUVOAHAJ1E3TXB1E8yhci/OGN2S2OzDl8W0HTKmuSG6
        Ab1ciZH0xC9XiF5aN3iRsCORW91DMYnmxn2zJZ19/0WpWnbpcT1VahT+SUoAKak5
        E5zEti5/b8x9iMRE6Yjje7IboJBLewIDAQABo1AwTjAdBgNVHQ4EFgQUqlhsaVZv
        0eq+02i/FG2sHRA6a44wHwYDVR0jBBgwFoAUqlhsaVZv0eq+02i/FG2sHRA6a44w
        DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEA2Iw4P01BpkiUfhO+vE2j
        QJ5eDD2KMFk6QKXs1Y+echOzjEulos32z7co6BBdeNd+3y09R2kCBGHPQ4IzA7rk
        dvwwAKCBqE6psPC5RC3ySi7Y5ANyGWfBJiXFrmdcSU8UiQvE7WLXoVHElggHzh0P
        UuM8zqIh1tlhCE0AAJy3uvkO38arp0oEpyP6Dzws61W6+0qLdPVU9VOS4faCQD5R
        J8x32LMb3K5K1pRfQbm5uq3skhE8+bPXGKqoe3SH0/CTNE7p0C1aSHmNBKsZDtNX
        SnuwFbwsJv2m8bWzNZVXRhRUGXy0qv6kdnLD8LMArLv0qtvcmkUBwWNuxNILWUKV
        tQ==
        -----END CERTIFICATE-----
      type: tls
    - addresses:
      - host: af1e2def1d7604343a759e72be5f643f-1329950098.us-east-2.elb.amazonaws.com
        port: 9094
      bootstrapServers: af1e2def1d7604343a759e72be5f643f-1329950098.us-east-2.elb.amazonaws.com:9094
      type: external
    observedGeneration: 5
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
prash911 commented 4 years ago

An update after disabling the tls it started working properly.

Not sure if it should be disabled. Should it be?

Commented following lines from template/deployment.yaml

#    tls:
#        trustedCertificates:
#            - secretName: {{ .Values.tls.certname }}
#              certificate: ca.crt
scholzj commented 4 years ago

Well, it is your choice really (consider security, versus performance etc.) - I would normally recommend using TLS unless you have some actual reasons against it.

But in any case, you can:

Both should work.

prash911 commented 4 years ago

Noted.

Thanks a lot for your help.

I am able to use both Tls and non tls ports with right config.