search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.86k stars 1.3k forks source link

Standalone Topic Operator can't connect via TLS without authentication #4766

Closed jozenstar closed 3 years ago

jozenstar commented 3 years ago

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug I'm using Strimzi to operate on AWS MSK cluster which requires TLS connection but doesn't have any authentication enabled at the moment. I've set the KafkaConnect successfully with this codeblock

tls:
    trustedCertificates: []

However, I can't set the Standalone Topic Operator to behave the same. If you set STRIMZI_TLS_ENABLED to true then the topic operator requires trust- and keystore locations and passwords to be set. If you set STRIMZI_TLS_ENABLED to false then you get the Java OOM (because of nonTLS connection). I've tried to set STRIMZI_TLS_ENABLED to false and set the security.protocol to SSL via STRIMZI_JAVA_SYSTEM_PROPERTIES, however this value is being overwritten by AdminConfig.

To Reproduce Steps to reproduce the behavior:

  1. Create an MSK cluster with strict TLS connection but without authentication
  2. Create a TopicOperator to connect to it
  3. Check logs
  4. See error

Expected behavior I'd like the standalone topic operator to be able to communicate via TLS without an authentication

Environment (please complete the following information):

YAML files and logs I've tried different configurations. I believe that this one should work, but it doesn't.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: strimzi-topic-operator
  labels:
    app: strimzi
  namespace: strimzi
spec:
  replicas: 1
  selector:
    matchLabels:
      name: strimzi-topic-operator
  template:
    metadata:
      labels:
        name: strimzi-topic-operator
    spec:
      serviceAccountName: strimzi-topic-operator
      containers:
        - name: strimzi-topic-operator
          image: quay.io/strimzi/operator:0.22.1
          args:
          - /opt/strimzi/bin/topic_operator_run.sh
          env:
            - name: STRIMZI_RESOURCE_LABELS
              value: "strimzi.io/cluster=my-cluster"
            - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS
              value: "Placeholder"
            - name: STRIMZI_ZOOKEEPER_CONNECT
              value: "Placeholder"
            - name: STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS
              value: "20000"
            - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
              value: "900000"
            - name: STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS
              value: "6"
            - name: STRIMZI_LOG_LEVEL
              value: DEBUG
            - name: STRIMZI_TLS_ENABLED
              value: "false"
            - name: STRIMZI_JAVA_SYSTEM_PROPERTIES
              value: "-Dsecurity.protocol=SSL"
            - name: STRIMZI_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /healthy
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30
          resources:
            limits:
              memory: 96Mi
              cpu: 100m
            requests:
              memory: 96Mi
              cpu: 100m
  strategy:
    type: Recreate
scholzj commented 3 years ago

TBH, using the operator outside of Strimzi in combination with something like MSK is not our main priority. So there was never any specific work done to support different authentication mechanisms. There is also no testing done for such use cases. So I think it is missing feature rather than a bug.

Have you tried to just provide a dummy keystore and truststore? I.e. provide something but without any actually certificates / keys. Is that something how it can be worked around?

jozenstar commented 3 years ago

@scholzj Thanks for the quick response!

Well, I've tried to create a dummy truststore from the java cacerts as it was described in this comment https://github.com/strimzi/strimzi-kafka-operator/issues/2761#issuecomment-609091174 However it didn't help.

Logs:

+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.topic-operator-0.22.1.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.7.jar:lib/io.fabric8.kubernetes-model-storageclass-5.0.2.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/io.fabric8.openshift-model-operator-5.0.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.60.Final.jar:lib/com.google.api.grpc.proto-google-common-protos-1.17.0.jar:lib/org.apache.kafka.connect-api-2.7.0.jar:lib/io.fabric8.kubernetes-model-settings-5.0.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.google.j2objc.j2objc-annotations-1.3.jar:lib/io.apicurio.apicurio-registry-common-1.3.0.Final.jar:lib/io.strimzi.api-0.22.1.jar:lib/io.fabric8.kubernetes-model-coordination-5.0.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.netty.netty-codec-4.1.60.Final.jar:lib/org.codehaus.mojo.animal-sniffer-annotations-1.18.jar:lib/io.netty.netty-codec-http2-4.1.60.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.0.2.jar:lib/io.grpc.grpc-context-1.31.1.jar:lib/io.fabric8.openshift-client-5.0.2.jar:lib/io.fabric8.openshift-model-console-5.0.2.jar:lib/io.grpc.grpc-protobuf-lite-1.31.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.0.Final.jar:lib/io.perfmark.perfmark-api-0.19.0.jar:lib/io.fabric8.kubernetes-model-rbac-5.0.2.jar:lib/io.fabric8.kubernetes-model-policy-5.0.2.jar:lib/io.netty.netty-codec-http-4.1.60.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.5.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.openshift-model-operatorhub-5.0.2.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.grpc.grpc-api-1.30.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/io.fabric8.kubernetes-model-discovery-5.0.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.5.1.jar:lib/io.fabric8.openshift-model-5.0.2.jar:lib/io.grpc.grpc-netty-shaded-1.31.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar:lib/io.fabric8.kubernetes-model-node-5.0.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.0.2.jar:lib/io.grpc.grpc-protobuf-1.31.1.jar:lib/com.google.android.annotations-4.1.1.4.jar:lib/io.fabric8.kubernetes-client-5.0.2.jar:lib/org.rocksdb.rocksdbjni-5.18.4.jar:lib/io.netty.netty-common-4.1.60.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.fabric8.kubernetes-model-apps-5.0.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/com.github.luben.zstd-jni-1.4.5-6.jar:lib/io.netty.netty-codec-socks-4.1.60.Final.jar:lib/io.netty.netty-transport-4.1.60.Final.jar:lib/io.strimzi.certificate-manager-0.22.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.0.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.60.Final.jar:lib/io.netty.netty-handler-proxy-4.1.60.Final.jar:lib/io.fabric8.openshift-model-monitoring-5.0.2.jar:lib/com.101tec.zkclient-0.11.jar:lib/com.google.guava.listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.checkerframework.checker-compat-qual-2.5.5.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.google.guava.failureaccess-1.0.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.5.jar:lib/io.fabric8.kubernetes-model-batch-5.0.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.kafka.kafka-clients-2.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.60.Final.jar:lib/io.netty.netty-buffer-4.1.60.Final.jar:lib/com.google.protobuf.protobuf-java-3.13.0.jar:lib/io.netty.netty-handler-4.1.60.Final.jar:lib/org.apache.kafka.kafka-streams-2.6.0.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.google.code.findbugs.jsr305-3.0.2.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.0.Final.jar:lib/io.fabric8.kubernetes-model-scheduling-5.0.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.5.jar:lib/io.fabric8.kubernetes-model-extensions-5.0.2.jar:lib/com.google.errorprone.error_prone_annotations-2.3.4.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/io.fabric8.kubernetes-model-events-5.0.2.jar:lib/io.grpc.grpc-core-1.31.1.jar:lib/io.strimzi.operator-common-0.22.1.jar:lib/com.google.guava.guava-28.2-android.jar:lib/io.fabric8.kubernetes-model-core-5.0.2.jar:lib/io.fabric8.kubernetes-model-networking-5.0.2.jar:lib/io.grpc.grpc-stub-1.31.1.jar:lib/io.fabric8.kubernetes-model-certificates-5.0.2.jar:lib/io.fabric8.kubernetes-model-common-5.0.2.jar:lib/io.netty.netty-resolver-4.1.60.Final.jar:lib/org.apache.kafka.connect-json-2.7.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.kubernetes-model-metrics-5.0.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.netty.netty-codec-dns-4.1.60.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.strimzi.crd-annotations-0.22.1.jar io.strimzi.operator.topic.Main
2021-04-16 14:48:12 INFO  Main:30 - TopicOperator 0.22.1 is starting
2021-04-16 14:48:13 DEBUG Config:519 - Trying to configure client from Kubernetes config...
2021-04-16 14:48:13 DEBUG Config:525 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2021-04-16 14:48:13 DEBUG Config:451 - Trying to configure client from service account...
2021-04-16 14:48:13 DEBUG Config:456 - Found service account host and port: 172.20.0.1:443
2021-04-16 14:48:13 DEBUG Config:462 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
2021-04-16 14:48:13 DEBUG Config:469 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2021-04-16 14:48:13 DEBUG Config:724 - Trying to configure client namespace from Kubernetes service account namespace path...
2021-04-16 14:48:13 DEBUG Config:729 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2021-04-16 14:48:21 DEBUG InternalLoggerFactory:63 - Using SLF4J as the default logging framework
2021-04-16 14:48:21 DEBUG ResourceLeakDetector:129 - -Dio.netty.leakDetection.level: simple
2021-04-16 14:48:21 DEBUG ResourceLeakDetector:130 - -Dio.netty.leakDetection.targetRecords: 4
2021-04-16 14:48:21 DEBUG InternalThreadLocalMap:83 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-04-16 14:48:21 DEBUG InternalThreadLocalMap:86 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-04-16 14:48:22 DEBUG MultithreadEventLoopGroup:44 - -Dio.netty.eventLoopThreads: 2
2021-04-16 14:48:22 DEBUG NioEventLoop:106 - -Dio.netty.noKeySetOptimization: false
2021-04-16 14:48:22 DEBUG NioEventLoop:107 - -Dio.netty.selectorAutoRebuildThreshold: 512
2021-04-16 14:48:22 DEBUG PlatformDependent0:417 - -Dio.netty.noUnsafe: false
2021-04-16 14:48:22 DEBUG PlatformDependent0:897 - Java version: 11
2021-04-16 14:48:22 DEBUG PlatformDependent0:130 - sun.misc.Unsafe.theUnsafe: available
2021-04-16 14:48:22 DEBUG PlatformDependent0:154 - sun.misc.Unsafe.copyMemory: available
2021-04-16 14:48:22 DEBUG PlatformDependent0:192 - java.nio.Buffer.address: available
2021-04-16 14:48:22 DEBUG PlatformDependent0:266 - direct buffer constructor: unavailable
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
        at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31) ~[io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:238) ~[io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:232) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:294) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:93) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:279) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:150) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:138) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:153) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.Vertx.vertx(Vertx.java:85) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.strimzi.operator.topic.Main.deploy(Main.java:50) [io.strimzi.topic-operator-0.22.1.jar:0.22.1]
        at io.strimzi.operator.topic.Main.run(Main.java:39) [io.strimzi.topic-operator-0.22.1.jar:0.22.1]
        at io.strimzi.operator.topic.Main.main(Main.java:32) [io.strimzi.topic-operator-0.22.1.jar:0.22.1]
2021-04-16 14:48:22 DEBUG PlatformDependent0:331 - java.nio.Bits.unaligned: available, true
2021-04-16 14:48:23 DEBUG PlatformDependent0:390 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @3f07b12c
        at jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:361) ~[?:?]
        at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:591) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:558) ~[?:?]
        at io.netty.util.internal.PlatformDependent0$6.run(PlatformDependent0.java:352) ~[io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:343) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:294) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:93) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:279) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:150) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:138) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:153) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.Vertx.vertx(Vertx.java:85) [io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.strimzi.operator.topic.Main.deploy(Main.java:50) [io.strimzi.topic-operator-0.22.1.jar:0.22.1]
        at io.strimzi.operator.topic.Main.run(Main.java:39) [io.strimzi.topic-operator-0.22.1.jar:0.22.1]
        at io.strimzi.operator.topic.Main.main(Main.java:32) [io.strimzi.topic-operator-0.22.1.jar:0.22.1]
2021-04-16 14:48:23 DEBUG PlatformDependent0:403 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
2021-04-16 14:48:23 DEBUG PlatformDependent:1080 - sun.misc.Unsafe: available
2021-04-16 14:48:23 DEBUG PlatformDependent:1182 - maxDirectMemory: 48693248 bytes (maybe)
2021-04-16 14:48:23 DEBUG PlatformDependent:1201 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2021-04-16 14:48:23 DEBUG PlatformDependent:1280 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2021-04-16 14:48:23 DEBUG PlatformDependent:178 - -Dio.netty.maxDirectMemory: -1 bytes
2021-04-16 14:48:23 DEBUG PlatformDependent:185 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2021-04-16 14:48:23 DEBUG CleanerJava9:71 - java.nio.ByteBuffer.cleaner(): available
2021-04-16 14:48:23 DEBUG PlatformDependent:205 - -Dio.netty.noPreferDirect: false
2021-04-16 14:48:23 DEBUG PlatformDependent:941 - org.jctools-core.MpscChunkedArrayQueue: available
2021-04-16 14:48:24 DEBUG InternalLoggerFactory:61 - Using SLF4J as the default logging framework
2021-04-16 14:48:26 DEBUG DefaultDnsServerAddressStreamProvider:80 - Default DNS servers: [/172.20.0.10:53] (sun.net.dns.ResolverConfiguration)
2021-04-16 14:48:26 DEBUG NetUtil:135 - -Djava.net.preferIPv4Stack: false
2021-04-16 14:48:26 DEBUG NetUtil:136 - -Djava.net.preferIPv6Addresses: false
2021-04-16 14:48:26 DEBUG NetUtilInitializations:129 - Loopback interface: lo (lo, 127.0.0.1)
2021-04-16 14:48:26 DEBUG NetUtil:169 - /proc/sys/net/core/somaxconn: 128
2021-04-16 14:48:27 INFO  Session:72 - Using config:
        STRIMZI_TRUSTSTORE_LOCATION: /tmp/topic-operator/topic-operator-truststore.p12
        STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=my-cluster
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 900000
        STRIMZI_CLIENT_ID: strimzi-topic-operator-a2041fe8-039f-4b79-9867-91b9b14ac4a1
        STRIMZI_STALE_RESULT_TIMEOUT_MS: 5000
        STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
        STRIMZI_KEYSTORE_LOCATION: /tmp/topic-operator/topic-operator-truststore.p12
        STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
        STRIMZI_USE_ZOOKEEPER_TOPIC_STORE: false
        STRIMZI_KAFKA_BOOTSTRAP_SERVERS: #PLACEHOLDER#
        STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
        STRIMZI_NAMESPACE: strimzi
        STRIMZI_APPLICATION_ID: __strimzi-topic-operator-kstreams
        STRIMZI_APPLICATION_SERVER: localhost:9000
        STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 20000
        STRIMZI_DISTRIBUTED_STORE: false
        STRIMZI_TOPICS_PATH: /strimzi/topics
        STRIMZI_ZOOKEEPER_CONNECT: #PLACEHOLDER#
        STRIMZI_TLS_ENABLED: true
        STRIMZI_KEYSTORE_PASSWORD: ********
        STRIMZI_STORE_NAME: topic-store
        STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
        TC_ZK_CONNECTION_TIMEOUT_MS: 20000
        STRIMZI_TRUSTSTORE_PASSWORD: ********
        STRIMZI_STORE_TOPIC: __strimzi_store_topic

2021-04-16 14:48:28 INFO  Session:149 - Starting
2021-04-16 14:48:28 INFO  AdminClientConfig:361 - AdminClientConfig values: 
        bootstrap.servers = [#PLACEHOLDER#]
        client.dns.lookup = use_all_dns_ips
        client.id = 
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = SSL
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 127000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = HTTPS
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = /tmp/topic-operator/topic-operator-truststore.p12
        ssl.keystore.password = [hidden]
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = /tmp/topic-operator/topic-operator-truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = JKS

2021-04-16 14:48:29 DEBUG AdminMetadataManager:245 - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [#PLACEHOLDER# (id: -1 rack: null), #PLACEHOLDER# (id: -2 rack: null), #PLACEHOLDER# (id: -3 rack: null)], partitions = [], controller = null).
Apr 16, 2021 2:48:31 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2900 ms, time limit is 2000 ms
Apr 16, 2021 2:48:32 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3899 ms, time limit is 2000 ms
Apr 16, 2021 2:48:33 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4899 ms, time limit is 2000 ms
Apr 16, 2021 2:48:34 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5999 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@11.0.10/java.security.MessageDigest.update(MessageDigest.java:355)
        at java.base@11.0.10/java.security.MessageDigest.digest(MessageDigest.java:430)
        at java.base@11.0.10/com.sun.crypto.provider.PKCS12PBECipherCore.derive(PKCS12PBECipherCore.java:119)
        at java.base@11.0.10/com.sun.crypto.provider.PKCS12PBECipherCore.derive(PKCS12PBECipherCore.java:69)
        at java.base@11.0.10/com.sun.crypto.provider.HmacPKCS12PBESHA1.engineInit(HmacPKCS12PBESHA1.java:134)
        at java.base@11.0.10/javax.crypto.Mac.chooseProvider(Mac.java:366)
        at java.base@11.0.10/javax.crypto.Mac.init(Mac.java:465)
        at java.base@11.0.10/sun.security.pkcs12.PKCS12KeyStore.lambda$engineLoad$2(PKCS12KeyStore.java:2151)
        at java.base@11.0.10/sun.security.pkcs12.PKCS12KeyStore$$Lambda$234/0x0000000840271040.tryOnce(Unknown Source)
        at java.base@11.0.10/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:295)
        at java.base@11.0.10/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2149)
        at java.base@11.0.10/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243)
        at java.base@11.0.10/java.security.KeyStore.load(KeyStore.java:1479)
        at app//org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374)
        at app//org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349)
        at app//org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:322)
        at app//org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:168)
        at app//org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:136)
        at app//org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:93)
        at app//org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        at app//org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at app//org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at app//org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at app//org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)
        at app//org.apache.kafka.clients.admin.Admin.create(Admin.java:65)
        at app//org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
        at app//io.strimzi.operator.topic.Session.start(Session.java:166)
        at app//io.vertx.core.impl.DeploymentManager.lambda$doDeploy$9(DeploymentManager.java:556)
        at app//io.vertx.core.impl.DeploymentManager$$Lambda$201/0x0000000840223c40.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
        at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
        at app//io.vertx.core.impl.EventLoopContext$$Lambda$202/0x0000000840240040.run(Unknown Source)
        at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

2021-04-16 14:48:35 DEBUG DefaultSslEngineFactory:264 - Created SSL context with keystore SecurityStore(path=/tmp/topic-operator/topic-operator-truststore.p12, modificationTime=Fri Apr 16 13:09:12 UTC 2021), truststore SecurityStore(path=/tmp/topic-operator/topic-operator-truststore.p12, modificationTime=Fri Apr 16 13:09:12 UTC 2021), provider SunJSSE.
Apr 16, 2021 2:48:35 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 6999 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@11.0.10/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base@11.0.10/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        at java.base@11.0.10/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
        at java.base@11.0.10/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base@11.0.10/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at app//org.apache.kafka.common.metrics.stats.Meter.stats(Meter.java:73)
        at app//org.apache.kafka.common.metrics.Sensor.add(Sensor.java:292)
        at app//org.apache.kafka.common.metrics.Sensor.add(Sensor.java:275)
        at app//org.apache.kafka.common.network.Selector$SelectorMetrics.<init>(Selector.java:1155)
        at app//org.apache.kafka.common.network.Selector.<init>(Selector.java:177)
        at app//org.apache.kafka.common.network.Selector.<init>(Selector.java:212)
        at app//org.apache.kafka.common.network.Selector.<init>(Selector.java:224)
        at app//org.apache.kafka.common.network.Selector.<init>(Selector.java:228)
        at app//org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:509)
        at app//org.apache.kafka.clients.admin.Admin.create(Admin.java:65)
        at app//org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
        at app//io.strimzi.operator.topic.Session.start(Session.java:166)
        at app//io.vertx.core.impl.DeploymentManager.lambda$doDeploy$9(DeploymentManager.java:556)
        at app//io.vertx.core.impl.DeploymentManager$$Lambda$201/0x0000000840223c40.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
        at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
        at app//io.vertx.core.impl.EventLoopContext$$Lambda$202/0x0000000840240040.run(Unknown Source)
        at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

2021-04-16 14:48:35 WARN  AdminClientConfig:369 - The configuration 'ssl.truststore.location' was supplied but isn't a known config.
2021-04-16 14:48:35 WARN  AdminClientConfig:369 - The configuration 'ssl.keystore.password' was supplied but isn't a known config.
2021-04-16 14:48:35 WARN  AdminClientConfig:369 - The configuration 'ssl.keystore.location' was supplied but isn't a known config.
2021-04-16 14:48:35 WARN  AdminClientConfig:369 - The configuration 'ssl.truststore.password' was supplied but isn't a known config.
2021-04-16 14:48:35 WARN  AdminClientConfig:369 - The configuration 'application.id' was supplied but isn't a known config.
2021-04-16 14:48:35 WARN  AdminClientConfig:369 - The configuration 'ssl.endpoint.identification.algorithm' was supplied but isn't a known config.
2021-04-16 14:48:35 INFO  AppInfoParser:119 - Kafka version: 2.7.0
2021-04-16 14:48:35 INFO  AppInfoParser:120 - Kafka commitId: 448719dc99a19793
2021-04-16 14:48:35 INFO  AppInfoParser:121 - Kafka startTimeMs: 1618584515748
2021-04-16 14:48:35 DEBUG KafkaAdminClient:586 - [AdminClient clientId=adminclient-1] Kafka admin client initialized
2021-04-16 14:48:35 DEBUG Session:167 - Using AdminClient org.apache.kafka.clients.admin.KafkaAdminClient@2251469a
2021-04-16 14:48:35 DEBUG Session:169 - Using Kafka io.strimzi.operator.topic.KafkaImpl@5a414534
2021-04-16 14:48:35 DEBUG Session:173 - Using namespace strimzi
2021-04-16 14:48:36 DEBUG NetworkClient:1003 - [AdminClient clientId=adminclient-1] Initiating connection to node #PLACEHOLDER# (id: -2 rack: null) using address #PLACEHOLDER#/#PLACEHOLDER#
Apr 16, 2021 2:48:36 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 7999 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@11.0.10/java.io.RandomAccessFile.seek0(Native Method)
        at java.base@11.0.10/java.io.RandomAccessFile.seek(RandomAccessFile.java:591)
        at java.base@11.0.10/java.util.zip.ZipFile$Source.readFullyAt(ZipFile.java:1314)
        at java.base@11.0.10/java.util.zip.ZipFile$ZipFileInputStream.initDataOffset(ZipFile.java:1003)
        at java.base@11.0.10/java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:1018)
        at java.base@11.0.10/java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:468)
        at java.base@11.0.10/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
        at java.base@11.0.10/jdk.internal.loader.Resource.getBytes(Resource.java:124)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:797)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
        at java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
        at java.base@11.0.10/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base@11.0.10/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at app//io.fabric8.kubernetes.api.model.apiextensions.v1beta1.CustomResourceDefinitionNamesBuilder.<init>(CustomResourceDefinitionNamesBuilder.java:25)
        at app//io.fabric8.kubernetes.api.model.apiextensions.v1beta1.CustomResourceDefinitionNamesBuilder.<init>(CustomResourceDefinitionNamesBuilder.java:21)
        at app//io.fabric8.kubernetes.api.model.apiextensions.v1beta1.CustomResourceDefinitionSpecFluentImpl$NamesNestedImpl.<init>(CustomResourceDefinitionSpecFluentImpl.java:646)
        at app//io.fabric8.kubernetes.api.model.apiextensions.v1beta1.CustomResourceDefinitionSpecFluentImpl.withNewNames(CustomResourceDefinitionSpecFluentImpl.java:267)
        at app//io.strimzi.api.kafka.Crds.crd(Crds.java:236)
        at app//io.strimzi.api.kafka.Crds.crd(Crds.java:95)
        at app//io.strimzi.api.kafka.Crds.kafkaTopic(Crds.java:322)
        at app//io.strimzi.operator.topic.K8sImpl.<init>(K8sImpl.java:42)
        at app//io.strimzi.operator.topic.Session.start(Session.java:174)
        at app//io.vertx.core.impl.DeploymentManager.lambda$doDeploy$9(DeploymentManager.java:556)
        at app//io.vertx.core.impl.DeploymentManager$$Lambda$201/0x0000000840223c40.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
        at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
        at app//io.vertx.core.impl.EventLoopContext$$Lambda$202/0x0000000840240040.run(Unknown Source)
        at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

2021-04-16 14:48:36 DEBUG Session:175 - Using k8s io.strimzi.operator.topic.K8sImpl@73b18b25
2021-04-16 14:48:36 DEBUG Session:178 - Using client-Id strimzi-topic-operator-a2041fe8-039f-4b79-9867-91b9b14ac4a1
2021-04-16 14:48:37 INFO  ZkEventThread:65 - Starting ZkClient event thread.
2021-04-16 14:48:37 DEBUG ZkConnection:69 - Creating new ZookKeeper instance to connect to #PLACEHOLDER#.
2021-04-16 14:48:37 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:host.name=strimzi-topic-operator-55c977d9dd-t7246
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.version=11.0.10
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.vendor=Red Hat, Inc.
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.10.0.9-1.el7_9.x86_64
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.class.path=lib/io.strimzi.topic-operator-0.22.1.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.7.jar:lib/io.fabric8.kubernetes-model-storageclass-5.0.2.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/io.fabric8.openshift-model-operator-5.0.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.60.Final.jar:lib/com.google.api.grpc.proto-google-common-protos-1.17.0.jar:lib/org.apache.kafka.connect-api-2.7.0.jar:lib/io.fabric8.kubernetes-model-settings-5.0.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.google.j2objc.j2objc-annotations-1.3.jar:lib/io.apicurio.apicurio-registry-common-1.3.0.Final.jar:lib/io.strimzi.api-0.22.1.jar:lib/io.fabric8.kubernetes-model-coordination-5.0.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.netty.netty-codec-4.1.60.Final.jar:lib/org.codehaus.mojo.animal-sniffer-annotations-1.18.jar:lib/io.netty.netty-codec-http2-4.1.60.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.0.2.jar:lib/io.grpc.grpc-context-1.31.1.jar:lib/io.fabric8.openshift-client-5.0.2.jar:lib/io.fabric8.openshift-model-console-5.0.2.jar:lib/io.grpc.grpc-protobuf-lite-1.31.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.0.Final.jar:lib/io.perfmark.perfmark-api-0.19.0.jar:lib/io.fabric8.kubernetes-model-rbac-5.0.2.jar:lib/io.fabric8.kubernetes-model-policy-5.0.2.jar:lib/io.netty.netty-codec-http-4.1.60.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.5.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.openshift-model-operatorhub-5.0.2.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.grpc.grpc-api-1.30.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/io.fabric8.kubernetes-model-discovery-5.0.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.5.1.jar:lib/io.fabric8.openshift-model-5.0.2.jar:lib/io.grpc.grpc-netty-shaded-1.31.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar:lib/io.fabric8.kubernetes-model-node-5.0.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.0.2.jar:lib/io.grpc.grpc-protobuf-1.31.1.jar:lib/com.google.android.annotations-4.1.1.4.jar:lib/io.fabric8.kubernetes-client-5.0.2.jar:lib/org.rocksdb.rocksdbjni-5.18.4.jar:lib/io.netty.netty-common-4.1.60.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.fabric8.kubernetes-model-apps-5.0.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/com.github.luben.zstd-jni-1.4.5-6.jar:lib/io.netty.netty-codec-socks-4.1.60.Final.jar:lib/io.netty.netty-transport-4.1.60.Final.jar:lib/io.strimzi.certificate-manager-0.22.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.0.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.60.Final.jar:lib/io.netty.netty-handler-proxy-4.1.60.Final.jar:lib/io.fabric8.openshift-model-monitoring-5.0.2.jar:lib/com.101tec.zkclient-0.11.jar:lib/com.google.guava.listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.checkerframework.checker-compat-qual-2.5.5.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.google.guava.failureaccess-1.0.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.5.jar:lib/io.fabric8.kubernetes-model-batch-5.0.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.kafka.kafka-clients-2.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.60.Final.jar:lib/io.netty.netty-buffer-4.1.60.Final.jar:lib/com.google.protobuf.protobuf-java-3.13.0.jar:lib/io.netty.netty-handler-4.1.60.Final.jar:lib/org.apache.kafka.kafka-streams-2.6.0.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.google.code.findbugs.jsr305-3.0.2.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.0.Final.jar:lib/io.fabric8.kubernetes-model-scheduling-5.0.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.5.jar:lib/io.fabric8.kubernetes-model-extensions-5.0.2.jar:lib/com.google.errorprone.error_prone_annotations-2.3.4.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/io.fabric8.kubernetes-model-events-5.0.2.jar:lib/io.grpc.grpc-core-1.31.1.jar:lib/io.strimzi.operator-common-0.22.1.jar:lib/com.google.guava.guava-28.2-android.jar:lib/io.fabric8.kubernetes-model-core-5.0.2.jar:lib/io.fabric8.kubernetes-model-networking-5.0.2.jar:lib/io.grpc.grpc-stub-1.31.1.jar:lib/io.fabric8.kubernetes-model-certificates-5.0.2.jar:lib/io.fabric8.kubernetes-model-common-5.0.2.jar:lib/io.netty.netty-resolver-4.1.60.Final.jar:lib/org.apache.kafka.connect-json-2.7.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.kubernetes-model-metrics-5.0.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.netty.netty-codec-dns-4.1.60.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.strimzi.crd-annotations-0.22.1.jar
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.io.tmpdir=/tmp
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:java.compiler=<NA>
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:os.name=Linux
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:os.arch=amd64
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:os.version=4.14.219-164.354.amzn2.x86_64
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:user.name=strimzi
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:user.home=/home/strimzi
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:user.dir=/opt/strimzi
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:os.memory.free=7MB
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:os.memory.max=46MB
2021-04-16 14:48:37 INFO  ZooKeeper:109 - Client environment:os.memory.total=24MB
2021-04-16 14:48:38 INFO  ZooKeeper:868 - Initiating client connection, connectString=#PLACEHOLDER# sessionTimeout=20000 watcher=org.I0Itec.zkclient.ZkClient@12b63229
2021-04-16 14:48:38 INFO  X509Util:79 - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2021-04-16 14:48:38 INFO  ClientCnxnSocket:237 - jute.maxbuffer value is 4194304 Bytes
2021-04-16 14:48:38 INFO  ClientCnxn:1653 - zookeeper.request.timeout value is 0. feature enabled=
2021-04-16 14:48:38 DEBUG ZkClient:1230 - Awaiting connection to Zookeeper server
2021-04-16 14:48:38 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
2021-04-16 14:48:38 DEBUG SaslServerPrincipal:80 - Canonicalized address to ip-10-16-53-128.eu-west-1.compute.internal
2021-04-16 14:48:38 INFO  ClientCnxn:1112 - Opening socket connection to server#PLACEHOLDER#/#PLACEHOLDER#. Will not attempt to authenticate using SASL (unknown error)
2021-04-16 14:48:42 DEBUG NetworkClient:955 - [AdminClient clientId=adminclient-1] Completed connection to node -2. Fetching API versions.
2021-04-16 14:48:44 DEBUG SslTransportLayer:452 - [SslTransportLayer channelId=-2 key=channel=java.nio.channels.SocketChannel[connection-pending remote=#PLACEHOLDER#/#PLACEHOLDER#], selector=sun.nio.ch.EPollSelectorImpl@7b062217, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost '#PLACEHOLDER#' peerPort 9094 peerPrincipal 'CN=*#PLACEHOLDER#' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384'
2021-04-16 14:48:44 DEBUG Selector:560 - [AdminClient clientId=adminclient-1] #PLACEHOLDER#/#PLACEHOLDER#
2021-04-16 14:48:44 DEBUG NetworkClient:969 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -2.
2021-04-16 14:48:44 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0) and timeout 3600000 to node -2: {client_software_name=apache-kafka-java,client_software_version=2.7.0,_tagged_fields={}}
2021-04-16 14:48:45 WARN  ClientCnxn:1190 - Client session timed out, have not heard from server in 6970ms for sessionid 0x0
2021-04-16 14:48:45 INFO  ClientCnxn:1238 - Client session timed out, have not heard from server in 6970ms for sessionid 0x0, closing socket connection and attempting reconnect
2021-04-16 14:48:45 DEBUG ClientCnxnSocketNIO:201 - Ignoring exception during shutdown input
java.net.SocketException: Socket is not connected
        at sun.nio.ch.Net.translateToSocketException(Net.java:162) ~[?:?]
        at sun.nio.ch.Net.translateException(Net.java:196) ~[?:?]
        at sun.nio.ch.Net.translateException(Net.java:202) ~[?:?]
        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:400) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1338) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1276) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1254) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
Caused by: java.nio.channels.NotYetConnectedException
        at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:917) ~[?:?]
        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:398) ~[?:?]
        ... 4 more
2021-04-16 14:48:45 DEBUG ClientCnxnSocketNIO:208 - Ignoring exception during shutdown output
java.net.SocketException: Socket is not connected
        at sun.nio.ch.Net.translateToSocketException(Net.java:162) ~[?:?]
        at sun.nio.ch.Net.translateException(Net.java:196) ~[?:?]
        at sun.nio.ch.Net.translateException(Net.java:202) ~[?:?]
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:408) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1338) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1276) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1254) [org.apache.zookeeper.zookeeper-3.5.8.jar:3.5.8]
Caused by: java.nio.channels.NotYetConnectedException
        at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:934) ~[?:?]
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:406) ~[?:?]
        ... 4 more
2021-04-16 14:48:45 DEBUG NetworkClient:884 - [AdminClient clientId=adminclient-1] Received API_VERSIONS response from node -2 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0): org.apache.kafka.common.requests.ApiVersionsResponse@363ccefd
2021-04-16 14:48:45 DEBUG NetworkClient:926 - [AdminClient clientId=adminclient-1] Recorded API versions for node -2: (Produce(0): 0 to 8 [usable: 8], Fetch(1): 0 to 12 [usable: 12], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 9], LeaderAndIsr(4): 0 to 4 [usable: 4], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 6 [usable: 6], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 7 [usable: 7], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 7 [usable: 7], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 6 [usable: 6], DeleteTopics(20): 0 to 5 [usable: 5], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 3 [usable: 3], AddPartitionsToTxn(24): 0 to 2 [usable: 2], AddOffsetsToTxn(25): 0 to 2 [usable: 2], EndTxn(26): 0 to 2 [usable: 2], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 2 [usable: 2], CreateAcls(30): 0 to 2 [usable: 2], DeleteAcls(31): 0 to 2 [usable: 2], DescribeConfigs(32): 0 to 3 [usable: 3], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 2 [usable: 2], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 2 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 [usable: 0], AlterClientQuotas(49): 0 [usable: 0], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterIsr(56): 0 [usable: 0], UpdateFeatures(57): 0 [usable: 0])
2021-04-16 14:48:45 DEBUG KafkaAdminClient:1094 - [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to #PLACEHOLDER# (id: -2 rack: null). correlationId=1
2021-04-16 14:48:45 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=9, clientId=adminclient-1, correlationId=1) and timeout 19899 to node -2: {topics=[],allow_auto_topic_creation=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false,_tagged_fields={}}
2021-04-16 14:48:46 DEBUG NetworkClient:884 - [AdminClient clientId=adminclient-1] Received METADATA response from node -2 for request with header RequestHeader(apiKey=METADATA, apiVersion=9, clientId=adminclient-1, correlationId=1): org.apache.kafka.common.requests.MetadataResponse@4fefd6d5
2021-04-16 14:48:46 DEBUG SaslServerPrincipal:80 - Canonicalized address to ip-10-16-109-109.eu-west-1.compute.internal
2021-04-16 14:48:46 INFO  ClientCnxn:1112 - Opening socket connection to server #PLACEHOLDER#/#PLACEHOLDER#. Will not attempt to authenticate using SASL (unknown error)
2021-04-16 14:48:46 DEBUG AdminMetadataManager:247 - [AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = ywVgm3tCTtyIncbviIcXBQ, nodes = [#PLACEHOLDER#(id: 3 rack: euw1-az3), #PLACEHOLDER# (id: 2 rack: euw1-az1), #PLACEHOLDER# (id: 1 rack: euw1-az2)], partitions = [], controller = #PLACEHOLDER# (id: 3 rack: euw1-az3))

UPD. Updated logs since previous were uncomplete.

jozenstar commented 3 years ago

@scholzj I've also found out that Topic Operator uses TLS v.1.3, whereas AWS MSK uses TLS v1.2. I've tried to specifiy -Dssl.protocol=TLSv1.2 with STRIMZI_JAVA_SYSTEM_PROPERTIES env var, but it was overwritten by the AdminConfig component. Is there a way to specify a property so it won't be overwritten by AdminConfig?

scholzj commented 3 years ago

This should be now fixed in #5201