Describe the bug
When I install the strimzi-kafka-operator (with helm) to a Specific namespace, logging in this case. It enters in a OOMKilled-CrashLoopBackOff loop apparently due to:
Warning Unhealthy 4m19s (x6 over 7m19s) kubelet Liveness probe failed: Get "http://172.17.0.3:8080/healthy": dial tcp 172.17.0.3:8080: connect: connection refused
Warning Unhealthy 4m19s (x11 over 7m19s) kubelet Readiness probe failed: Get "http://172.17.0.3:8080/ready": dial tcp 172.17.0.3:8080: connect: connection refused
Normal Killing 4m19s (x2 over 6m19s) kubelet Container strimzi-cluster-operator failed liveness probe, will be restarted
Warning BackOff 2m37s (x4 over 3m9s) kubelet Back-off restarting failed container
I've tried passing no value overrides and passing the following values, I get the same result.
Expected behavior
The strimzi-cluster-operator- pod starts just fine
Environment (please complete the following information):
Strimzi version: [e.g. main, 0.26.1]
Installation method: [Helm chart]
Kubernetes cluster: [v1.22.3]
Infrastructure: [minikube v1.24.0 on Darwin 11.5.2 (arm64)]
YAML files and logs
Attach or copy paste the custom resources
pod describe
Name: strimzi-cluster-operator-86cdffd6d7-9996p
Namespace: logging
Priority: 0
Node: minikube/192.168.49.2
Start Time: Wed, 22 Dec 2021 20:38:58 -0500
Labels: name=strimzi-cluster-operator
pod-template-hash=86cdffd6d7
strimzi.io/kind=cluster-operator
Annotations: <none>
Status: Running
IP: 172.17.0.3
IPs:
IP: 172.17.0.3
Controlled By: ReplicaSet/strimzi-cluster-operator-86cdffd6d7
Containers:
strimzi-cluster-operator:
Container ID: docker://0ef126d8c06a65bd340857aac54816de8486c23c2336d3d1f045ff525ddce005
Image: quay.io/strimzi/operator:0.26.1
Image ID: docker-pullable://quay.io/strimzi/operator@sha256:5d90caa66f57939ba62a9a50efa00fce319ba596e719a3f8a42170d7635ee3f4
Port: 8080/TCP
Host Port: 0/TCP
Args:
/opt/strimzi/bin/cluster_operator_run.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Wed, 22 Dec 2021 20:44:29 -0500
Finished: Wed, 22 Dec 2021 20:44:58 -0500
Ready: False
Restart Count: 5
Limits:
cpu: 1
memory: 384Mi
Requests:
cpu: 200m
memory: 384Mi
Liveness: http-get http://:http/healthy delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://:http/ready delay=10s timeout=1s period=30s #success=1 #failure=3
Environment:
STRIMZI_NAMESPACE: logging (v1:metadata.namespace)
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
STRIMZI_OPERATION_TIMEOUT_MS: 300000
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_CONNECT_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.0
STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.26.1
STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.26.1
STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.26.1
STRIMZI_OPERATOR_NAMESPACE: logging (v1:metadata.namespace)
STRIMZI_FEATURE_GATES:
Mounts:
/opt/strimzi/custom-config/ from co-config-volume (rw)
/tmp from strimzi-tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8pbpw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
strimzi-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 1Mi
co-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: strimzi-cluster-operator
Optional: false
kube-api-access-8pbpw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m56s default-scheduler Successfully assigned logging/strimzi-cluster-operator-86cdffd6d7-9996p to minikube
Normal Pulling 6m55s kubelet Pulling image "quay.io/strimzi/operator:0.26.1"
Normal Pulled 6m36s kubelet Successfully pulled image "quay.io/strimzi/operator:0.26.1" in 18.780064175s
Warning Unhealthy 5m56s kubelet Liveness probe failed: Get "http://172.17.0.3:8080/healthy": read tcp 172.17.0.1:55832->172.17.0.3:8080: read: connection reset by peer
Warning Unhealthy 5m56s kubelet Readiness probe failed: Get "http://172.17.0.3:8080/ready": read tcp 172.17.0.1:55834->172.17.0.3:8080: read: connection reset by peer
Warning Unhealthy 5m26s kubelet Readiness probe failed: Get "http://172.17.0.3:8080/ready": read tcp 172.17.0.1:56020->172.17.0.3:8080: read: connection reset by peer
Warning Unhealthy 5m26s kubelet Liveness probe failed: Get "http://172.17.0.3:8080/healthy": read tcp 172.17.0.1:56018->172.17.0.3:8080: read: connection reset by peer
Normal Created 4m33s (x4 over 6m34s) kubelet Created container strimzi-cluster-operator
Normal Started 4m33s (x4 over 6m34s) kubelet Started container strimzi-cluster-operator
Normal Pulled 4m33s (x3 over 5m55s) kubelet Container image "quay.io/strimzi/operator:0.26.1" already present on machine
Warning Unhealthy 3m56s kubelet Liveness probe failed: Get "http://172.17.0.3:8080/healthy": read tcp 172.17.0.1:56510->172.17.0.3:8080: read: connection reset by peer
Normal Killing 3m56s kubelet Container strimzi-cluster-operator failed liveness probe, will be restarted
Warning Unhealthy 3m56s kubelet Readiness probe failed: Get "http://172.17.0.3:8080/ready": read tcp 172.17.0.1:56512->172.17.0.3:8080: read: connection reset by peer
Warning Unhealthy 3m56s kubelet Readiness probe failed: Get "http://172.17.0.3:8080/ready": dial tcp 172.17.0.3:8080: connect: connection refused
Warning BackOff 110s (x11 over 5m25s) kubelet Back-off restarting failed container
pod logs
Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom '
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.26.1.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-policy-5.8.0.jar:lib/io.fabric8.kubernetes-model-batch-5.8.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.8.0.jar:lib/io.fabric8.openshift-model-machineconfig-5.8.0.jar:lib/io.fabric8.kubernetes-model-certificates-5.8.0.jar:lib/io.fabric8.kubernetes-model-node-5.8.0.jar:lib/io.netty.netty-handler-proxy-4.1.68.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-codec-socks-4.1.68.Final.jar:lib/io.netty.netty-codec-dns-4.1.68.Final.jar:lib/io.netty.netty-transport-4.1.68.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-5.8.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.strimzi.certificate-manager-0.26.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.8.0.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.68.Final.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.strimzi.api-0.26.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.8.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.fabric8.openshift-model-operatorhub-5.8.0.jar:lib/io.fabric8.kubernetes-model-apps-5.8.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.15.0.jar:lib/io.fabric8.openshift-model-console-5.8.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.8.0.jar:lib/org.apache.logging.log4j.log4j-api-2.15.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.8.0.jar:lib/io.netty.netty-resolver-4.1.68.Final.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.fabric8.kubernetes-model-events-5.8.0.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.fabric8.kubernetes-model-core-5.8.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.8.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.8.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.5.jar:lib/io.netty.netty-codec-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.8.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-model-5.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/io.fabric8.openshift-client-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.fabric8.kubernetes-model-rbac-5.8.0.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.netty.netty-handler-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-server-plain-0.8.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.8.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.8.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.kubernetes-client-5.8.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.8.0.jar:lib/io.vertx.vertx-core-4.1.5.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-http-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-client-0.8.1.jar:lib/io.fabric8.openshift-model-operator-5.8.0.jar:lib/io.fabric8.kubernetes-model-common-5.8.0.jar:lib/io.fabric8.openshift-model-tuned-5.8.0.jar:lib/io.strimzi.config-model-0.26.1.jar:lib/io.netty.netty-common-4.1.68.Final.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.netty.netty-buffer-4.1.68.Final.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.logging.log4j.log4j-core-2.15.0.jar:lib/io.strimzi.kafka-oauth-server-0.8.1.jar:lib/io.strimzi.kafka-oauth-common-0.8.1.jar:lib/io.fabric8.openshift-model-monitoring-5.8.0.jar:lib/io.strimzi.crd-annotations-0.26.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-storageclass-5.8.0.jar:lib/io.fabric8.openshift-model-machine-5.8.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-http2-4.1.68.Final.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-networking-5.8.0.jar:lib/io.strimzi.operator-common-0.26.1.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
2021-12-23 01:44:38 INFO Main:60 - ClusterOperator 0.26.1 is starting
2021-12-23 01:44:40 INFO Main:62 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[logging],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{2.8.0={proto: 2.8 msg: 2.8 kafka-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0 connect-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0 mirrormaker-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0 mirrormaker2-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0}, 2.8.1={proto: 2.8 msg: 2.8 kafka-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1 connect-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1 mirrormaker-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1 mirrormaker2-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1}, 3.0.0={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0 connect-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0 mirrormaker-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0 mirrormaker2-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0}},imagePullPolicy=null,imagePullSecrets=null,operatorNamespace=logging,operatorNamespaceLabels=null,rbacScope=CLUSTER,customResourceSelector=null,featureGates=FeatureGates(controlPlaneListener=false,ServiceAccountPatching=false))
2021-12-23 01:44:50 INFO Main:81 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.22,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false,OpenShiftDeploymentConfigs=false)
2021-12-23 01:44:50 INFO Util:303 - Using config:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container: oci
STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.0
KUBERNETES_PORT: tcp://10.96.0.1:443
STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
JAVA_HOME: /usr/lib/jvm/jre-11
JAVA_OPTS: -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
KUBERNETES_SERVICE_HOST: 10.96.0.1
JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.26.1.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-policy-5.8.0.jar:lib/io.fabric8.kubernetes-model-batch-5.8.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.8.0.jar:lib/io.fabric8.openshift-model-machineconfig-5.8.0.jar:lib/io.fabric8.kubernetes-model-certificates-5.8.0.jar:lib/io.fabric8.kubernetes-model-node-5.8.0.jar:lib/io.netty.netty-handler-proxy-4.1.68.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-codec-socks-4.1.68.Final.jar:lib/io.netty.netty-codec-dns-4.1.68.Final.jar:lib/io.netty.netty-transport-4.1.68.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-5.8.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.strimzi.certificate-manager-0.26.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.8.0.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.68.Final.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.strimzi.api-0.26.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.8.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.fabric8.openshift-model-operatorhub-5.8.0.jar:lib/io.fabric8.kubernetes-model-apps-5.8.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.15.0.jar:lib/io.fabric8.openshift-model-console-5.8.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.8.0.jar:lib/org.apache.logging.log4j.log4j-api-2.15.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.8.0.jar:lib/io.netty.netty-resolver-4.1.68.Final.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.fabric8.kubernetes-model-events-5.8.0.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.fabric8.kubernetes-model-core-5.8.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.8.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.8.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.5.jar:lib/io.netty.netty-codec-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.8.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-model-5.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/io.fabric8.openshift-client-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.fabric8.kubernetes-model-rbac-5.8.0.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.netty.netty-handler-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-server-plain-0.8.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.8.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.8.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.kubernetes-client-5.8.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.8.0.jar:lib/io.vertx.vertx-core-4.1.5.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-http-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-client-0.8.1.jar:lib/io.fabric8.openshift-model-operator-5.8.0.jar:lib/io.fabric8.kubernetes-model-common-5.8.0.jar:lib/io.fabric8.openshift-model-tuned-5.8.0.jar:lib/io.strimzi.config-model-0.26.1.jar:lib/io.netty.netty-common-4.1.68.Final.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.netty.netty-buffer-4.1.68.Final.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.logging.log4j.log4j-core-2.15.0.jar:lib/io.strimzi.kafka-oauth-server-0.8.1.jar:lib/io.strimzi.kafka-oauth-common-0.8.1.jar:lib/io.fabric8.openshift-model-monitoring-5.8.0.jar:lib/io.strimzi.crd-annotations-0.26.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-storageclass-5.8.0.jar:lib/io.fabric8.openshift-model-machine-5.8.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-http2-4.1.68.Final.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-networking-5.8.0.jar:lib/io.strimzi.operator-common-0.26.1.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar
STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_HOME: /opt/strimzi
PWD: /opt/strimzi
TINI_VERSION: v0.19.0
KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
STRIMZI_OPERATION_TIMEOUT_MS: 300000
JAVA_MAIN: io.strimzi.operator.cluster.Main
KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
STRIMZI_VERSION: 0.26.1
STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.26.1
STRIMZI_NAMESPACE: logging
STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.26.1
STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.26.1
KUBERNETES_PORT_443_TCP_PROTO: tcp
STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_FEATURE_GATES:
KUBERNETES_SERVICE_PORT: 443
STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.26.1
TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
HOSTNAME: strimzi-cluster-operator-86cdffd6d7-9996p
STRIMZI_KAFKA_CONNECT_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
SHLVL: 0
HOME: /home/strimzi
MALLOC_ARENA_MAX: 2
STRIMZI_OPERATOR_NAMESPACE: logging
2021-12-23 01:44:50 INFO ClusterOperator:78 - Creating ClusterOperator for namespace logging
2021-12-23 01:44:50 INFO ClusterOperator:94 - Starting ClusterOperator for namespace logging
2021-12-23 01:44:51 INFO ClusterOperator:105 - Opened watch for Kafka operator
2021-12-23 01:44:51 INFO ClusterOperator:105 - Opened watch for KafkaMirrorMaker operator
2021-12-23 01:44:51 INFO ClusterOperator:105 - Opened watch for KafkaConnect operator
2021-12-23 01:44:52 INFO ClusterOperator:105 - Opened watch for KafkaBridge operator
2021-12-23 01:44:52 INFO ClusterOperator:105 - Opened watch for KafkaMirrorMaker2 operator
2021-12-23 01:44:52 INFO ClusterOperator:116 - Setting up periodic reconciliation for namespace logging
2021-12-23 01:44:53 INFO ClusterOperator:174 - ClusterOperator is now ready (health server listening on 8080)
2021-12-23 01:44:53 INFO Main:150 - Cluster Operator verticle started in namespace logging without label selector
Additional context
edit:
The same replication steps in EKS produce a healthy operator
k get no
NAME STATUS ROLES AGE VERSION
ip-10-0-1-34.ec2.internal Ready <none> 94m v1.20.11-eks-f17b81
ip-10-0-1-88.ec2.internal Ready <none> 93m v1.20.11-eks-f17b81
ip-10-0-3-246.ec2.internal Ready <none> 94m v1.20.11-eks-f17b81
k get po -n logging -w
NAME READY STATUS RESTARTS AGE
strimzi-cluster-operator-6b8d5b69c7-cfbmn 0/1 Running 0 22s
strimzi-cluster-operator-6b8d5b69c7-cfbmn 1/1 Running 0 40s
pod description
Name: strimzi-cluster-operator-6b8d5b69c7-cfbmn
Namespace: logging
Priority: 0
Node: ip-10-0-1-88.ec2.internal/10.0.1.88
Start Time: Wed, 22 Dec 2021 20:58:37 -0500
Labels: name=strimzi-cluster-operator
pod-template-hash=6b8d5b69c7
strimzi.io/kind=cluster-operator
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.0.1.44
IPs:
IP: 10.0.1.44
Controlled By: ReplicaSet/strimzi-cluster-operator-6b8d5b69c7
Containers:
strimzi-cluster-operator:
Container ID: docker://9e29c2d38010c39b9798a7f68998a48b42f433c68bfbfd5c85a1c32a8bdd6b8b
Image: quay.io/strimzi/operator:0.26.1
Image ID: docker-pullable://quay.io/strimzi/operator@sha256:5d90caa66f57939ba62a9a50efa00fce319ba596e719a3f8a42170d7635ee3f4
Port: 8080/TCP
Host Port: 0/TCP
Args:
/opt/strimzi/bin/cluster_operator_run.sh
State: Running
Started: Wed, 22 Dec 2021 20:58:47 -0500
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 384Mi
Requests:
cpu: 200m
memory: 384Mi
Liveness: http-get http://:http/healthy delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://:http/ready delay=10s timeout=1s period=30s #success=1 #failure=3
Environment:
STRIMZI_NAMESPACE: logging (v1:metadata.namespace)
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
STRIMZI_OPERATION_TIMEOUT_MS: 300000
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_CONNECT_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.0
STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.26.1
STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.26.1
STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.26.1
STRIMZI_OPERATOR_NAMESPACE: logging (v1:metadata.namespace)
STRIMZI_FEATURE_GATES:
Mounts:
/opt/strimzi/custom-config/ from co-config-volume (rw)
/tmp from strimzi-tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from strimzi-cluster-operator-token-2hlsw (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
strimzi-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 1Mi
co-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: strimzi-cluster-operator
Optional: false
strimzi-cluster-operator-token-2hlsw:
Type: Secret (a volume populated by a Secret)
SecretName: strimzi-cluster-operator-token-2hlsw
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m41s default-scheduler Successfully assigned logging/strimzi-cluster-operator-6b8d5b69c7-cfbmn to ip-10-0-1-88.ec2.internal
Normal Pulling 4m41s kubelet Pulling image "quay.io/strimzi/operator:0.26.1"
Normal Pulled 4m33s kubelet Successfully pulled image "quay.io/strimzi/operator:0.26.1" in 7.785920914s
Normal Created 4m31s kubelet Created container strimzi-cluster-operator
Normal Started 4m31s kubelet Started container strimzi-cluster-operator
pod logs
Follow Logs:
Show timestamp:
Since:
0
Tail:
-1
To terminal:
Filter:
Enter your key words
Wrap lines:
Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom '
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.26.1.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-policy-5.8.0.jar:lib/io.fabric8.kubernetes-model-batch-5.8.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.8.0.jar:lib/io.fabric8.openshift-model-machineconfig-5.8.0.jar:lib/io.fabric8.kubernetes-model-certificates-5.8.0.jar:lib/io.fabric8.kubernetes-model-node-5.8.0.jar:lib/io.netty.netty-handler-proxy-4.1.68.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-codec-socks-4.1.68.Final.jar:lib/io.netty.netty-codec-dns-4.1.68.Final.jar:lib/io.netty.netty-transport-4.1.68.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-5.8.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.strimzi.certificate-manager-0.26.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.8.0.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.68.Final.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.strimzi.api-0.26.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.8.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.fabric8.openshift-model-operatorhub-5.8.0.jar:lib/io.fabric8.kubernetes-model-apps-5.8.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.15.0.jar:lib/io.fabric8.openshift-model-console-5.8.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.8.0.jar:lib/org.apache.logging.log4j.log4j-api-2.15.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.8.0.jar:lib/io.netty.netty-resolver-4.1.68.Final.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.fabric8.kubernetes-model-events-5.8.0.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.fabric8.kubernetes-model-core-5.8.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.8.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.8.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.5.jar:lib/io.netty.netty-codec-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.8.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-model-5.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/io.fabric8.openshift-client-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.fabric8.kubernetes-model-rbac-5.8.0.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.netty.netty-handler-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-server-plain-0.8.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.8.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.8.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.kubernetes-client-5.8.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.8.0.jar:lib/io.vertx.vertx-core-4.1.5.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-http-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-client-0.8.1.jar:lib/io.fabric8.openshift-model-operator-5.8.0.jar:lib/io.fabric8.kubernetes-model-common-5.8.0.jar:lib/io.fabric8.openshift-model-tuned-5.8.0.jar:lib/io.strimzi.config-model-0.26.1.jar:lib/io.netty.netty-common-4.1.68.Final.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.netty.netty-buffer-4.1.68.Final.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.logging.log4j.log4j-core-2.15.0.jar:lib/io.strimzi.kafka-oauth-server-0.8.1.jar:lib/io.strimzi.kafka-oauth-common-0.8.1.jar:lib/io.fabric8.openshift-model-monitoring-5.8.0.jar:lib/io.strimzi.crd-annotations-0.26.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-storageclass-5.8.0.jar:lib/io.fabric8.openshift-model-machine-5.8.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-http2-4.1.68.Final.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-networking-5.8.0.jar:lib/io.strimzi.operator-common-0.26.1.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
2021-12-23 01:58:49 INFO Main:60 - ClusterOperator 0.26.1 is starting
2021-12-23 01:58:49 INFO Main:62 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[logging],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{2.8.0={proto: 2.8 msg: 2.8 kafka-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0 connect-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0 mirrormaker-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0 mirrormaker2-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.0}, 2.8.1={proto: 2.8 msg: 2.8 kafka-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1 connect-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1 mirrormaker-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1 mirrormaker2-image: quay.io/strimzi/kafka:0.26.1-kafka-2.8.1}, 3.0.0={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0 connect-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0 mirrormaker-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0 mirrormaker2-image: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0}},imagePullPolicy=null,imagePullSecrets=null,operatorNamespace=logging,operatorNamespaceLabels=null,rbacScope=CLUSTER,customResourceSelector=null,featureGates=FeatureGates(controlPlaneListener=false,ServiceAccountPatching=false))
2021-12-23 01:58:52 INFO Main:81 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.20,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false,OpenShiftDeploymentConfigs=false)
2021-12-23 01:58:52 INFO Util:303 - Using config:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container: oci
STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.0
KUBERNETES_PORT: tcp://172.20.0.1:443
STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
JAVA_HOME: /usr/lib/jvm/jre-11
JAVA_OPTS: -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
KUBERNETES_SERVICE_HOST: 172.20.0.1
JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.26.1.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-policy-5.8.0.jar:lib/io.fabric8.kubernetes-model-batch-5.8.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.8.0.jar:lib/io.fabric8.openshift-model-machineconfig-5.8.0.jar:lib/io.fabric8.kubernetes-model-certificates-5.8.0.jar:lib/io.fabric8.kubernetes-model-node-5.8.0.jar:lib/io.netty.netty-handler-proxy-4.1.68.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-codec-socks-4.1.68.Final.jar:lib/io.netty.netty-codec-dns-4.1.68.Final.jar:lib/io.netty.netty-transport-4.1.68.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-5.8.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.strimzi.certificate-manager-0.26.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.8.0.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.68.Final.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.strimzi.api-0.26.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.8.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.fabric8.openshift-model-operatorhub-5.8.0.jar:lib/io.fabric8.kubernetes-model-apps-5.8.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.15.0.jar:lib/io.fabric8.openshift-model-console-5.8.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.8.0.jar:lib/org.apache.logging.log4j.log4j-api-2.15.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.8.0.jar:lib/io.netty.netty-resolver-4.1.68.Final.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.fabric8.kubernetes-model-events-5.8.0.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.fabric8.kubernetes-model-core-5.8.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.8.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.8.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.5.jar:lib/io.netty.netty-codec-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.8.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-model-5.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/io.fabric8.openshift-client-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.fabric8.kubernetes-model-rbac-5.8.0.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.netty.netty-handler-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-server-plain-0.8.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.8.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.8.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.kubernetes-client-5.8.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.8.0.jar:lib/io.vertx.vertx-core-4.1.5.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-http-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-client-0.8.1.jar:lib/io.fabric8.openshift-model-operator-5.8.0.jar:lib/io.fabric8.kubernetes-model-common-5.8.0.jar:lib/io.fabric8.openshift-model-tuned-5.8.0.jar:lib/io.strimzi.config-model-0.26.1.jar:lib/io.netty.netty-common-4.1.68.Final.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.netty.netty-buffer-4.1.68.Final.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.logging.log4j.log4j-core-2.15.0.jar:lib/io.strimzi.kafka-oauth-server-0.8.1.jar:lib/io.strimzi.kafka-oauth-common-0.8.1.jar:lib/io.fabric8.openshift-model-monitoring-5.8.0.jar:lib/io.strimzi.crd-annotations-0.26.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-storageclass-5.8.0.jar:lib/io.fabric8.openshift-model-machine-5.8.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-http2-4.1.68.Final.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-networking-5.8.0.jar:lib/io.strimzi.operator-common-0.26.1.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar
STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_HOME: /opt/strimzi
PWD: /opt/strimzi
TINI_VERSION: v0.19.0
KUBERNETES_PORT_443_TCP: tcp://172.20.0.1:443
STRIMZI_OPERATION_TIMEOUT_MS: 300000
JAVA_MAIN: io.strimzi.operator.cluster.Main
KUBERNETES_PORT_443_TCP_ADDR: 172.20.0.1
STRIMZI_VERSION: 0.26.1
STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.26.1
STRIMZI_NAMESPACE: logging
STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.26.1
STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.26.1
KUBERNETES_PORT_443_TCP_PROTO: tcp
STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.26.1
STRIMZI_FEATURE_GATES:
KUBERNETES_SERVICE_PORT: 443
STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.26.1
TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
HOSTNAME: strimzi-cluster-operator-6b8d5b69c7-cfbmn
STRIMZI_KAFKA_CONNECT_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
STRIMZI_KAFKA_IMAGES: 2.8.0=quay.io/strimzi/kafka:0.26.1-kafka-2.8.0
2.8.1=quay.io/strimzi/kafka:0.26.1-kafka-2.8.1
3.0.0=quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.26.1-kafka-3.0.0
SHLVL: 0
HOME: /home/strimzi
MALLOC_ARENA_MAX: 2
STRIMZI_OPERATOR_NAMESPACE: logging
2021-12-23 01:58:52 INFO ClusterOperator:78 - Creating ClusterOperator for namespace logging
2021-12-23 01:58:52 INFO ClusterOperator:94 - Starting ClusterOperator for namespace logging
2021-12-23 01:58:52 INFO ClusterOperator:105 - Opened watch for Kafka operator
2021-12-23 01:58:52 INFO ClusterOperator:105 - Opened watch for KafkaMirrorMaker operator
2021-12-23 01:58:52 INFO ClusterOperator:105 - Opened watch for KafkaConnect operator
2021-12-23 01:58:52 INFO ClusterOperator:105 - Opened watch for KafkaBridge operator
2021-12-23 01:58:52 INFO ClusterOperator:105 - Opened watch for KafkaMirrorMaker2 operator
2021-12-23 01:58:52 INFO ClusterOperator:116 - Setting up periodic reconciliation for namespace logging
2021-12-23 01:58:52 INFO ClusterOperator:174 - ClusterOperator is now ready (health server listening on 8080)
2021-12-23 01:58:52 INFO Main:150 - Cluster Operator verticle started in namespace logging without label selector
2021-12-23 02:00:52 INFO ClusterOperator:118 - Triggering periodic reconciliation for namespace logging
2021-12-23 02:00:52 WARN VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2021-12-23 02:00:53 WARN VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2021-12-23 02:00:53 WARN VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2021-12-23 02:00:53 WARN VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2021-12-23 02:00:54 WARN VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2021-12-23 02:00:54 WARN VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
2021-12-23 02:02:52 INFO ClusterOperator:118 - Triggering periodic reconciliation for namespace logging
Describe the bug When I install the strimzi-kafka-operator (with helm) to a Specific namespace, logging in this case. It enters in a OOMKilled-CrashLoopBackOff loop apparently due to:
I've tried passing no value overrides and passing the following values, I get the same result.
To Reproduce Steps to reproduce the behavior:
kubectl create ns logging
helm install logging-stack -n logging strimzi/strimzi-kafka-operator
kubectl get po -n logging
Expected behavior The
strimzi-cluster-operator-
pod starts just fineEnvironment (please complete the following information):
YAML files and logs
Attach or copy paste the custom resources pod describe
pod logs
Additional context edit: The same replication steps in EKS produce a healthy operator
pod description
pod logs