search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.84k stars 1.29k forks source link

Strimzi recovery failure with Velero #3743

Closed cloudcafetech closed 4 years ago

cloudcafetech commented 4 years ago

After restore using velero

Below details ...

[root@sync-cluster setup-files]# oc get po
NAME                                            READY   STATUS             RESTARTS   AGE
akhq-ddcfcfd8b-5ftqn                            1/1     Running            0          5m10s
kafka-acl-viewer-f79b7fff6-gzgdw                1/1     Running            4          5m10s
strimzi-cluster-operator-68b6d59f74-4pz7c       0/1     CrashLoopBackOff   5          5m10s
sync-bridge-bridge-78bf5cf75f-k2vvh             1/1     Running            1          5m10s
sync-cluster-entity-operator-59b97b8c88-zlfrw   1/3     CrashLoopBackOff   11         5m10s
sync-cluster-kafka-0                            2/2     Running            1          5m9s
sync-cluster-kafka-1                            2/2     Running            1          5m9s
sync-cluster-kafka-2                            2/2     Running            1          5m9s
sync-cluster-zookeeper-0                        1/1     Running            0          5m9s
sync-cluster-zookeeper-1                        1/1     Running            0          5m9s
sync-cluster-zookeeper-2                        1/1     Running            0          5m9s

[root@sync-cluster setup-files]# oc logs -f strimzi-cluster-operator-68b6d59f74-4pz7c
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ java -version
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.kafka-oauth-client-0.5.0.jar:lib/io.strimzi.api-0.19.0.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.keycloak.keycloak-core-10.0.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/io.strimzi.config-model-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.keycloak.keycloak-common-10.0.0.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-common-0.5.0.jar:lib/io.strimzi.kafka-oauth-server-0.5.0.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.62.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
2020-10-04 02:25:30 INFO  Main:60 - ClusterOperator 0.19.0 is starting
2020-10-04 02:25:33 INFO  Main:86 - Environment facts gathered: ClusterOperatorConfig(KubernetesVersion=1.19,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false,OpenShiftDeploymentConfigs=false)
2020-10-04 02:25:33 INFO  Util:268 - Using config:
        PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        SYNC_CLUSTER_KAFKA_2_SERVICE_PORT: 9094
        STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: strimzi/operator:0.19.0
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP: tcp://10.96.164.152:9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP: tcp://10.96.142.104:9094
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT: tcp://10.96.164.152:9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
        KAFKA_ACL_VIEWER_SERVICE_HOST: 10.96.208.142
        STRIMZI_DEFAULT_TLS_SIDECAR_KAFKA_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT: tcp://10.96.98.125:9091
        STRIMZI_HOME: /opt/strimzi
        PWD: /opt/strimzi
        SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT_REST_API: 8080
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP: tcp://10.96.38.96:8080
        JAVA_MAIN: io.strimzi.operator.cluster.Main
        AKHQ_PORT_80_TCP_PORT: 80
        STRIMZI_DEFAULT_JMXTRANS_IMAGE: strimzi/jmxtrans:0.19.0
        STRIMZI_NAMESPACE: streaming
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_HOST: 10.96.164.152
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PROTO: tcp
        KAFKA_ACL_VIEWER_PORT: tcp://10.96.208.142:80
        KAFKA_ACL_VIEWER_PORT_80_TCP: tcp://10.96.208.142:80
        STRIMZI_LOG_LEVEL: INFO
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
        SYNC_CLUSTER_KAFKA_1_SERVICE_PORT: 9094
        KUBERNETES_SERVICE_PORT_HTTPS: 443
        AKHQ_PORT: tcp://10.96.4.192:80
        SHLVL: 0
        AKHQ_PORT_80_TCP: tcp://10.96.4.192:80
        AKHQ_SERVICE_PORT: 80
        KAFKA_ACL_VIEWER_PORT_80_TCP_ADDR: 10.96.208.142
        TINI_SHA256: 12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
        AKHQ_SERVICE_PORT_HTTP: 80
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP: tcp://10.96.151.227:9094
        STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0

        KUBERNETES_PORT: tcp://10.96.0.1:443
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.96.98.125
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
        JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.kafka-oauth-client-0.5.0.jar:lib/io.strimzi.api-0.19.0.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.keycloak.keycloak-core-10.0.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/io.strimzi.config-model-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.keycloak.keycloak-common-10.0.0.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-common-0.5.0.jar:lib/io.strimzi.kafka-oauth-server-0.5.0.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.62.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar
        SYNC_CLUSTER_KAFKA_2_SERVICE_HOST: 10.96.151.227
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.96.16.2:2181
        SYNC_CLUSTER_KAFKA_0_SERVICE_HOST: 10.96.142.104
        STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.96.98.125:9092
        KAFKA_ACL_VIEWER_PORT_80_TCP_PROTO: tcp
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.96.16.2
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_ADDR: 10.96.151.227
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PORT: 8080
        KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_ADDR: 10.96.100.188
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_ADDR: 10.96.142.104
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP: tcp://10.96.100.188:9094
        STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0

        KUBERNETES_PORT_443_TCP_PROTO: tcp
        STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: strimzi/operator:0.19.0
        STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: strimzi/operator:0.19.0
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
        STRIMZI_KAFKA_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0

        SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
        STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_ADDR: 10.96.164.152
        SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT: 8080
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PROTO: tcp
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
        SYNC_CLUSTER_KAFKA_2_PORT: tcp://10.96.151.227:9094
        SYNC_CLUSTER_KAFKA_1_PORT: tcp://10.96.100.188:9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.96.98.125
        SYNC_CLUSTER_KAFKA_1_SERVICE_HOST: 10.96.100.188
        KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT: tcp://10.96.16.2:2181
        STRIMZI_VERSION: 0.19.0
        SYNC_CLUSTER_KAFKA_0_PORT: tcp://10.96.142.104:9094
        AKHQ_PORT_80_TCP_PROTO: tcp
        AKHQ_SERVICE_HOST: 10.96.4.192
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT: 9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PROTO: tcp
        AKHQ_PORT_80_TCP_ADDR: 10.96.4.192
        SYNC_CLUSTER_KAFKA_0_SERVICE_PORT: 9094
        STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: strimzi/kafka-bridge:0.18.0
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT: tcp://10.96.38.96:8080
        SYNC_CLUSTER_KAFKA_0_SERVICE_PORT_TCP_EXTERNAL: 9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.96.98.125
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PROTO: tcp
        KUBERNETES_SERVICE_HOST: 10.96.0.1
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.96.98.125:9093
        SYNC_CLUSTER_KAFKA_1_SERVICE_PORT_TCP_EXTERNAL: 9094
        STRIMZI_KAFKA_CONNECT_S2I_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0

        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PORT: 9094
        SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_HOST: 10.96.38.96
        TINI_VERSION: v0.18.0
        STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
        KAFKA_ACL_VIEWER_SERVICE_PORT_80_TCP: 80
        STRIMZI_OPERATION_TIMEOUT_MS: 300000
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.96.98.125
        KAFKA_ACL_VIEWER_SERVICE_PORT: 80
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PORT: 9094
        KUBERNETES_SERVICE_PORT: 443
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.96.98.125:9091
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PORT: 9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PORT: 9094
        STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.96.16.2
        HOSTNAME: strimzi-cluster-operator-68b6d59f74-4pz7c
        STRIMZI_KAFKA_CONNECT_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0

        KUBERNETES_PORT_443_TCP_PORT: 443
        KAFKA_ACL_VIEWER_PORT_80_TCP_PORT: 80
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_ADDR: 10.96.38.96
        SYNC_CLUSTER_KAFKA_2_SERVICE_PORT_TCP_EXTERNAL: 9094
        HOME: /home/strimzi
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT_TCP_EXTERNAL: 9094
        MALLOC_ARENA_MAX: 2

2020-10-04 02:25:33 INFO  Main:127 - The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.
2020-10-04 02:25:33 INFO  ClusterOperator:81 - Creating ClusterOperator for namespace streaming
2020-10-04 02:25:33 INFO  ClusterOperator:98 - Starting ClusterOperator for namespace streaming
2020-10-04 02:25:34 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkas.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkas" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:34 INFO  AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:34 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormakers.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkamirrormakers" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:34 INFO  AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:35 WARN  WatchConnectionManager:366 - Executor didn't terminate in time after shutdown in close(), killing it in: io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@59c09998
2020-10-04 02:25:35 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkaconnects.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkaconnects" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:35 INFO  AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:35 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkabridges.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkabridges" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:35 INFO  AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:36 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormaker2s.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkamirrormaker2s" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:36 INFO  AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:36 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkaconnectors.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkaconnectors" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:36 WARN  WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkarebalances.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkarebalances" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:36 ERROR Main:162 - Cluster Operator verticle in namespace streaming failed to start
io.fabric8.kubernetes.client.KubernetesClientException: kafkas.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkas" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:206) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
        Suppressed: java.lang.Throwable: waiting here
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:135) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:354) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:759) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:738) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:69) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:63) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
                at io.strimzi.operator.common.AbstractOperator.lambda$createWatch$9(AbstractOperator.java:293) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
                at io.strimzi.operator.common.Util.lambda$async$0(Util.java:49) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
                at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
                at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
                at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
                at java.lang.Thread.run(Thread.java:834) [?:?]

[root@sync-cluster setup-files]# oc describe po sync-cluster-entity-operator-59b97b8c88-zlfrw
Name:         sync-cluster-entity-operator-59b97b8c88-zlfrw
Namespace:    streaming
Priority:     0
Node:         kind-control-plane/172.17.0.2
Start Time:   Sun, 04 Oct 2020 02:21:43 +0000
Labels:       app.kubernetes.io/instance=sync-cluster
              app.kubernetes.io/managed-by=strimzi-cluster-operator
              app.kubernetes.io/name=entity-operator
              app.kubernetes.io/part-of=strimzi-sync-cluster
              pod-template-hash=59b97b8c88
              strimzi.io/cluster=sync-cluster
              strimzi.io/kind=Kafka
              strimzi.io/name=sync-cluster-entity-operator
              velero.io/backup-name=velero-bkp-sync-cluster.streaming.full-04-10-2020-02-14-43
              velero.io/restore-name=velero-restore-sync-cluster-04-10-2020-02-21-40
Annotations:  strimzi.io/clients-ca-cert-generation: 0
              strimzi.io/cluster-ca-cert-generation: 0
Status:       Running
IP:           10.244.0.27
IPs:
  IP:           10.244.0.27
Controlled By:  ReplicaSet/sync-cluster-entity-operator-59b97b8c88
Containers:
  topic-operator:
    Container ID:  containerd://b51a51f9a72c2c3c8db61cf9e8aa96ad235880688b0682076a768c4de3bf49d5
    Image:         strimzi/operator:0.19.0
    Image ID:      docker.io/strimzi/operator@sha256:d8fed9eb28c6d024670b060527681e21c6744a6c7dd2bf48450ce577382d3646
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      /opt/strimzi/bin/topic_operator_run.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 04 Oct 2020 02:29:23 +0000
      Finished:     Sun, 04 Oct 2020 02:29:54 +0000
    Ready:          False
    Restart Count:  7
    Liveness:       http-get http://:healthcheck/healthy delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:      http-get http://:healthcheck/ready delay=10s timeout=5s period=10s #success=1 #failure=3
    Environment:
      STRIMZI_RESOURCE_LABELS:                  strimzi.io/cluster=sync-cluster
      STRIMZI_KAFKA_BOOTSTRAP_SERVERS:          sync-cluster-kafka-bootstrap:9091
      STRIMZI_ZOOKEEPER_CONNECT:                localhost:2181
      STRIMZI_NAMESPACE:                        streaming
      STRIMZI_FULL_RECONCILIATION_INTERVAL_MS:  90000
      STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS:     20000
      STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS:      6
      STRIMZI_TLS_ENABLED:                      true
      STRIMZI_GC_LOG_ENABLED:                   false
    Mounts:
      /etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
      /etc/tls-sidecar/eo-certs/ from eo-certs (rw)
      /opt/topic-operator/custom-config/ from entity-topic-operator-metrics-and-logging (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from sync-cluster-entity-operator-token-s75jc (ro)
  user-operator:
    Container ID:  containerd://dcdbe68ba4c33004d3cfa8b0ccdf6861df3cf2d8f51a65fed34b5976b9047d6e
    Image:         strimzi/operator:0.19.0
    Image ID:      docker.io/strimzi/operator@sha256:d8fed9eb28c6d024670b060527681e21c6744a6c7dd2bf48450ce577382d3646
    Port:          8081/TCP
    Host Port:     0/TCP
    Args:
      /opt/strimzi/bin/user_operator_run.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 04 Oct 2020 02:27:57 +0000
      Finished:     Sun, 04 Oct 2020 02:27:59 +0000
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:healthcheck/healthy delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:      http-get http://:healthcheck/ready delay=10s timeout=5s period=10s #success=1 #failure=3
    Environment:
      STRIMZI_ZOOKEEPER_CONNECT:                localhost:2181
      STRIMZI_KAFKA_BOOTSTRAP_SERVERS:          sync-cluster-kafka-bootstrap:9091
      STRIMZI_NAMESPACE:                        streaming
      STRIMZI_LABELS:                           strimzi.io/cluster=sync-cluster
      STRIMZI_FULL_RECONCILIATION_INTERVAL_MS:  120000
      STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS:     6000
      STRIMZI_CA_KEY_NAME:                      sync-cluster-clients-ca
      STRIMZI_CA_CERT_NAME:                     sync-cluster-clients-ca-cert
      STRIMZI_CA_NAMESPACE:                     streaming
      STRIMZI_CA_VALIDITY:                      365
      STRIMZI_CA_RENEWAL:                       30
      STRIMZI_CLUSTER_CA_CERT_SECRET_NAME:      sync-cluster-cluster-ca-cert
      STRIMZI_EO_KEY_SECRET_NAME:               sync-cluster-entity-operator-certs
      STRIMZI_GC_LOG_ENABLED:                   false
    Mounts:
      /etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
      /etc/tls-sidecar/eo-certs/ from eo-certs (rw)
      /opt/user-operator/custom-config/ from entity-user-operator-metrics-and-logging (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from sync-cluster-entity-operator-token-s75jc (ro)
  tls-sidecar:
    Container ID:  containerd://9619d9e30ad1c5c87f94ee94862a41025f19e71890a1a52b0189d05ba2076401
    Image:         strimzi/kafka:0.19.0-kafka-2.5.0
    Image ID:      docker.io/strimzi/kafka@sha256:961419d795036f5f491817a5652c8f30c531a7b3cafd245d9473812a6a632c19
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/stunnel/entity_operator_stunnel_run.sh
    State:          Running
      Started:      Sun, 04 Oct 2020 02:21:46 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3
    Readiness:      exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3
    Environment:
      TLS_SIDECAR_LOG_LEVEL:      notice
      STRIMZI_ZOOKEEPER_CONNECT:  sync-cluster-zookeeper-client:2181
    Mounts:
      /etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
      /etc/tls-sidecar/eo-certs/ from eo-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from sync-cluster-entity-operator-token-s75jc (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  entity-topic-operator-metrics-and-logging:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sync-cluster-entity-topic-operator-config
    Optional:  false
  entity-user-operator-metrics-and-logging:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sync-cluster-entity-user-operator-config
    Optional:  false
  eo-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sync-cluster-entity-operator-certs
    Optional:    false
  cluster-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sync-cluster-cluster-ca-cert
    Optional:    false
  sync-cluster-entity-operator-token-s75jc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sync-cluster-entity-operator-token-s75jc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  9m55s                   default-scheduler  Successfully assigned streaming/sync-cluster-entity-operator-59b97b8c88-zlfrw to kind-control-plane
  Normal   Created    9m53s                   kubelet            Created container topic-operator
  Normal   Started    9m53s                   kubelet            Started container topic-operator
  Normal   Started    9m52s                   kubelet            Started container tls-sidecar
  Normal   Created    9m52s                   kubelet            Created container tls-sidecar
  Normal   Pulled     9m52s                   kubelet            Container image "strimzi/kafka:0.19.0-kafka-2.5.0" already present on machine
  Normal   Started    9m20s (x3 over 9m52s)   kubelet            Started container user-operator
  Normal   Created    9m20s (x3 over 9m53s)   kubelet            Created container user-operator
  Normal   Pulled     9m20s (x3 over 9m53s)   kubelet            Container image "strimzi/operator:0.19.0" already present on machine
  Warning  Unhealthy  9m16s (x3 over 9m36s)   kubelet            Readiness probe failed: Get "http://10.244.0.27:8080/ready": dial tcp 10.244.0.27:8080: connect: connection refused
  Normal   Pulled     9m14s (x2 over 9m53s)   kubelet            Container image "strimzi/operator:0.19.0" already present on machine
  Warning  Unhealthy  9m14s (x3 over 9m34s)   kubelet            Liveness probe failed: Get "http://10.244.0.27:8080/healthy": dial tcp 10.244.0.27:8080: connect: connection refused
  Normal   Killing    9m14s                   kubelet            Container topic-operator failed liveness probe, will be restarted
  Warning  BackOff    4m49s (x36 over 9m35s)  kubelet            Back-off restarting failed container
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]# oc logs -f sync-cluster-entity-operator-59b97b8c88-zlfrw -c topic-operator
Preparing certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/ca.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca
Certificate was added to keystore
Preparing certificates for internal communication is complete
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' false == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ java -version
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.topic-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.topic.Main
[2020-10-04 02:29:24,965] INFO  <Main        :31> [main        ] TopicOperator 0.19.0 is starting
[2020-10-04 02:29:25,597] INFO  <Session     :66> [main        ] Using config:
        STRIMZI_TRUSTSTORE_LOCATION: /tmp/topic-operator/replication.truststore.p12
        STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=sync-cluster
        STRIMZI_KAFKA_BOOTSTRAP_SERVERS: sync-cluster-kafka-bootstrap:9091
        STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
        STRIMZI_NAMESPACE: streaming
        STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 20000
        STRIMZI_TOPICS_PATH: /strimzi/topics
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 90000
        STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
        STRIMZI_TLS_ENABLED: true
        STRIMZI_KEYSTORE_PASSWORD: ********
        STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
        STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
        STRIMZI_KEYSTORE_LOCATION: /tmp/topic-operator/replication.keystore.p12
        TC_ZK_CONNECTION_TIMEOUT_MS: 20000
        STRIMZI_TRUSTSTORE_PASSWORD: ********
        STRIMZI_REASSIGN_THROTTLE: 9223372036854775807

[2020-10-04 02:29:25,619] INFO  <Session     :135> [oop-thread-0] Starting
[2020-10-04 02:29:25,646] INFO  <ClientConfig:347> [oop-thread-0] AdminClientConfig values:
        bootstrap.servers = [sync-cluster-kafka-bootstrap:9091]
        client.dns.lookup = default
        client.id =
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = SSL
        security.providers = null
        send.buffer.bytes = 131072
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2]
        ssl.endpoint.identification.algorithm = HTTPS
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = /tmp/topic-operator/replication.keystore.p12
        ssl.keystore.password = [hidden]
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.2
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = /tmp/topic-operator/replication.truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = JKS

[2020-10-04 02:29:25,868] WARN  <ClientConfig:355> [oop-thread-0] The configuration 'ssl.truststore.location' was supplied but isn't a known config.
[2020-10-04 02:29:25,868] WARN  <ClientConfig:355> [oop-thread-0] The configuration 'ssl.keystore.password' was supplied but isn't a known config.
[2020-10-04 02:29:25,869] WARN  <ClientConfig:355> [oop-thread-0] The configuration 'ssl.keystore.location' was supplied but isn't a known config.
[2020-10-04 02:29:25,869] WARN  <ClientConfig:355> [oop-thread-0] The configuration 'ssl.truststore.password' was supplied but isn't a known config.
[2020-10-04 02:29:25,869] WARN  <ClientConfig:355> [oop-thread-0] The configuration 'ssl.endpoint.identification.algorithm' was supplied but isn't a known config.
[2020-10-04 02:29:25,876] INFO  <ppInfoParser:117> [oop-thread-0] Kafka version: 2.5.0
[2020-10-04 02:29:25,877] INFO  <ppInfoParser:118> [oop-thread-0] Kafka commitId: 66563e712b0b9f84
[2020-10-04 02:29:25,878] INFO  <ppInfoParser:119> [oop-thread-0] Kafka startTimeMs: 1601778565869
[2020-10-04 02:29:25,956] INFO  <kEventThread:65> [calhost:2181] Starting ZkClient event thread.
[2020-10-04 02:29:25,969] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
[2020-10-04 02:29:25,969] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:host.name=sync-cluster-entity-operator-59b97b8c88-zlfrw
[2020-10-04 02:29:25,969] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.version=11.0.7
[2020-10-04 02:29:25,970] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.vendor=Oracle Corporation
[2020-10-04 02:29:25,971] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.7.10-4.el7_8.x86_64
[2020-10-04 02:29:25,971] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.class.path=lib/io.strimzi.topic-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar
[2020-10-04 02:29:25,972] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
[2020-10-04 02:29:25,972] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.io.tmpdir=/tmp
[2020-10-04 02:29:25,973] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:java.compiler=<NA>
[2020-10-04 02:29:25,973] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:os.name=Linux
[2020-10-04 02:29:25,973] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:os.arch=amd64
[2020-10-04 02:29:25,974] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
[2020-10-04 02:29:25,974] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:user.name=strimzi
[2020-10-04 02:29:25,974] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:user.home=/home/strimzi
[2020-10-04 02:29:25,974] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:user.dir=/opt/strimzi
[2020-10-04 02:29:25,975] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:os.memory.free=172MB
[2020-10-04 02:29:25,975] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:os.memory.max=3596MB
[2020-10-04 02:29:25,975] INFO  <ZooKeeper   :109> [ker-thread-0] Client environment:os.memory.total=226MB
[2020-10-04 02:29:25,979] INFO  <ZooKeeper   :868> [ker-thread-0] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.I0Itec.zkclient.ZkClient@4f20ffb5
[2020-10-04 02:29:25,986] INFO  <X509Util    :79> [ker-thread-0] Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
[2020-10-04 02:29:25,992] INFO  <ntCnxnSocket:237> [ker-thread-0] jute.maxbuffer value is 4194304 Bytes
[2020-10-04 02:29:26,047] INFO  <ClientCnxn  :1653> [ker-thread-0] zookeeper.request.timeout value is 0. feature enabled=
[2020-10-04 02:29:26,050] INFO  <ZkClient    :936> [ker-thread-0] Waiting for keeper state SyncConnected
[2020-10-04 02:29:26,070] INFO  <ClientCnxn  :1112> [alhost:2181)] Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
[2020-10-04 02:29:26,071] INFO  <ClientCnxn  :1244> [alhost:2181)] Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused
[2020-10-04 02:29:27,174] INFO  <ClientCnxn  :1112> [alhost:2181)] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[2020-10-04 02:29:27,175] INFO  <ClientCnxn  :959> [alhost:2181)] Socket connection established, initiating session, client: /127.0.0.1:52458, server: localhost/127.0.0.1:2181
[2020-10-04 02:29:27,229] INFO  <ClientCnxn  :1394> [alhost:2181)] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x3000012a6080003, negotiated timeout = 20000
[2020-10-04 02:29:27,232] INFO  <ZkClient    :713> [-EventThread] zookeeper state changed (SyncConnected)
[2020-10-04 02:29:27,261] INFO  <opicOperator:1215> [oop-thread-0] Starting initial  reconciliation
[2020-10-04 02:29:27,274] INFO  <Session     :232> [oop-thread-0] Started
[2020-10-04 02:29:27,613] WARN  <onUsageUtils:60> [s-ops-tool-0] The client is using resource type 'kafkatopics' with unstable version 'v1beta1'
[2020-10-04 02:29:27,801] WARN  <ctionManager:202> [0.96.0.1/...] Exec Failure: HTTP 403, Status: 403 - kafkatopics.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot watch resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,806] ERROR <opicOperator:1351> [oop-thread-1] 0|initial kafka shipments: Error getting KafkaTopic shipments for topic shipments
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/streaming/kafkatopics/shipments. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkatopics.kafka.strimzi.io "shipments" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,820] ERROR <opicOperator:1351> [oop-thread-1] 1|initial kafka sales: Error getting KafkaTopic sales for topic sales
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/streaming/kafkatopics/sales. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkatopics.kafka.strimzi.io "sales" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,821] ERROR <opicOperator:1351> [oop-thread-1] 2|initial kafka users: Error getting KafkaTopic users for topic users
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/streaming/kafkatopics/users. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkatopics.kafka.strimzi.io "users" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,824] ERROR <Main        :63> [oop-thread-1] Error deploying Session
io.fabric8.kubernetes.client.KubernetesClientException: kafkatopics.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot watch resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:206) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
        Suppressed: java.lang.Throwable: waiting here
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:135) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:354) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:759) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:738) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:69) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
                at io.strimzi.operator.topic.Session.lambda$start$3(Session.java:197) ~[io.strimzi.topic-operator-0.19.0.jar:0.19.0]
                at java.lang.Thread.run(Thread.java:834) [?:?]
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]# oc logs -f sync-cluster-entity-operator-59b97b8c88-zlfrw -c user-operator
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' false == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ java -version
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.user-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.10.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.user.Main
[2020-10-04 02:27:58,314] INFO  <Main        :54> [main        ] UserOperator 0.19.0 is starting
[2020-10-04 02:27:58,957] INFO  <Util        :268> [main        ] Using config:
        PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        STRIMZI_CA_RENEWAL: 30
        SYNC_CLUSTER_KAFKA_2_SERVICE_PORT: 9094
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP: tcp://10.96.164.152:9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP: tcp://10.96.142.104:9094
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT: tcp://10.96.164.152:9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
        KAFKA_ACL_VIEWER_SERVICE_HOST: 10.96.208.142
        STRIMZI_EO_KEY_SECRET_NAME: sync-cluster-entity-operator-certs
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT: tcp://10.96.98.125:9091
        STRIMZI_HOME: /opt/strimzi
        PWD: /opt/strimzi
        SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT_REST_API: 8080
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP: tcp://10.96.38.96:8080
        JAVA_MAIN: io.strimzi.operator.user.Main
        AKHQ_PORT_80_TCP_PORT: 80
        STRIMZI_NAMESPACE: streaming
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_HOST: 10.96.164.152
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PROTO: tcp
        STRIMZI_CA_VALIDITY: 365
        KAFKA_ACL_VIEWER_PORT: tcp://10.96.208.142:80
        KAFKA_ACL_VIEWER_PORT_80_TCP: tcp://10.96.208.142:80
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
        SYNC_CLUSTER_KAFKA_1_SERVICE_PORT: 9094
        KUBERNETES_SERVICE_PORT_HTTPS: 443
        AKHQ_PORT: tcp://10.96.4.192:80
        SHLVL: 0
        AKHQ_PORT_80_TCP: tcp://10.96.4.192:80
        AKHQ_SERVICE_PORT: 80
        KAFKA_ACL_VIEWER_PORT_80_TCP_ADDR: 10.96.208.142
        TINI_SHA256: 12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
        AKHQ_SERVICE_PORT_HTTP: 80
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP: tcp://10.96.151.227:9094
        KUBERNETES_PORT: tcp://10.96.0.1:443
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.96.98.125
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
        JAVA_CLASSPATH: lib/io.strimzi.user-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.10.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar
        SYNC_CLUSTER_KAFKA_2_SERVICE_HOST: 10.96.151.227
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.96.16.2:2181
        SYNC_CLUSTER_KAFKA_0_SERVICE_HOST: 10.96.142.104
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.96.98.125:9092
        KAFKA_ACL_VIEWER_PORT_80_TCP_PROTO: tcp
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.96.16.2
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_ADDR: 10.96.151.227
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PORT: 8080
        KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_ADDR: 10.96.100.188
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_ADDR: 10.96.142.104
        STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP: tcp://10.96.100.188:9094
        KUBERNETES_PORT_443_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_ADDR: 10.96.164.152
        SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT: 8080
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PROTO: tcp
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
        STRIMZI_LABELS: strimzi.io/cluster=sync-cluster
        JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny
        SYNC_CLUSTER_KAFKA_2_PORT: tcp://10.96.151.227:9094
        SYNC_CLUSTER_KAFKA_1_PORT: tcp://10.96.100.188:9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.96.98.125
        STRIMZI_GC_LOG_ENABLED: false
        SYNC_CLUSTER_KAFKA_1_SERVICE_HOST: 10.96.100.188
        KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT: tcp://10.96.16.2:2181
        STRIMZI_VERSION: 0.19.0
        STRIMZI_KAFKA_BOOTSTRAP_SERVERS: sync-cluster-kafka-bootstrap:9091
        SYNC_CLUSTER_KAFKA_0_PORT: tcp://10.96.142.104:9094
        AKHQ_PORT_80_TCP_PROTO: tcp
        AKHQ_SERVICE_HOST: 10.96.4.192
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT: 9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PROTO: tcp
        STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
        AKHQ_PORT_80_TCP_ADDR: 10.96.4.192
        SYNC_CLUSTER_KAFKA_0_SERVICE_PORT: 9094
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT: tcp://10.96.38.96:8080
        SYNC_CLUSTER_KAFKA_0_SERVICE_PORT_TCP_EXTERNAL: 9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.96.98.125
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PROTO: tcp
        KUBERNETES_SERVICE_HOST: 10.96.0.1
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.96.98.125:9093
        SYNC_CLUSTER_KAFKA_1_SERVICE_PORT_TCP_EXTERNAL: 9094
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PORT: 9094
        STRIMZI_CLUSTER_CA_CERT_SECRET_NAME: sync-cluster-cluster-ca-cert
        SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_HOST: 10.96.38.96
        TINI_VERSION: v0.18.0
        KAFKA_ACL_VIEWER_SERVICE_PORT_80_TCP: 80
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.96.98.125
        KAFKA_ACL_VIEWER_SERVICE_PORT: 80
        STRIMZI_CA_CERT_NAME: sync-cluster-clients-ca-cert
        STRIMZI_CA_NAMESPACE: streaming
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
        SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PORT: 9094
        KUBERNETES_SERVICE_PORT: 443
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.96.98.125:9091
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PORT: 9094
        SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
        SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PORT: 9094
        SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.96.16.2
        HOSTNAME: sync-cluster-entity-operator-59b97b8c88-zlfrw
        STRIMZI_CA_KEY_NAME: sync-cluster-clients-ca
        KUBERNETES_PORT_443_TCP_PORT: 443
        KAFKA_ACL_VIEWER_PORT_80_TCP_PORT: 80
        SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_ADDR: 10.96.38.96
        SYNC_CLUSTER_KAFKA_2_SERVICE_PORT_TCP_EXTERNAL: 9094
        HOME: /home/strimzi
        SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT_TCP_EXTERNAL: 9094
        SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PROTO: tcp
        MALLOC_ARENA_MAX: 2

[2020-10-04 02:27:59,624] ERROR <Main        :75> [oop-thread-0] Unable to start operator
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/streaming/secrets/sync-cluster-cluster-ca-cert. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets "sync-cluster-cluster-ca-cert" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "secrets" in API group "" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
scholzj commented 4 years ago

I think the error messages are quite clear - the operators do not have the RBAC rights they need to work. So whatever you did to backup and restore, it did not worked properly and lost the RBAC settings which you would need to fix.

cloudcafetech commented 4 years ago

Hmm, looks like issue in taking backup with cluster resources (CRD,clusterrole etc.) ....

Will try and update u ...

cloudcafetech commented 4 years ago

Able to recover but little complex and need to maintained sequence ...

Personally I like velero for kubernetes backup and restore with persistent data .... Not yet tested with high volume data.

zarelit commented 3 years ago

@cloudcafetech you closed the issue, what was the resolution for it?

cloudcafetech commented 3 years ago

Restoration should have proper sequence. Please follow restore docs.

psavva commented 3 years ago

@cloudcafetech Can you advise on your restore procedure? I am reviewing velero for my backups/restore solution, and would also need to backup/restore strimzi kafka operator.

What sequence are you following? which is the "proper sequence"?

I have backed up the kafka namespace, which holds the strimzi operator. Then I have backed up the namespace which holds my kafka cluster: ie: cluster namespace Third, I have restored kafka namespace, then restored cluster namespace.

cloudcafetech commented 3 years ago

Sequence as follows ...

PVC Cluster Operator Topics Kafka Cluster Users

psavva commented 3 years ago

@cloudcafetech Thank you :)

veerendra2 commented 2 years ago

@cloudcafetech Does backup also follow sequence? Can paste velero restore commands to restore in mentioned sequence?