Reconciliation #0(watch) Kafka(proj/kafka): createOrUpdate failed

[cameronbraid]

version 0.6.0

2018-11-01 07:27:44 ERROR AbstractAssemblyOperator:244 - Reconciliation #0(watch) Kafka(proj/kafka): createOrUpdate failed
--
  | io.fabric8.kubernetes.client.KubernetesClientException: Got unexpected POST status code 403: Forbidden
  | at io.strimzi.operator.common.operator.resource.WorkaroundRbacOperator.execute(WorkaroundRbacOperator.java:118) ~[cluster-operator-0.6.0.jar:?]
  | at io.strimzi.operator.common.operator.resource.WorkaroundRbacOperator.create(WorkaroundRbacOperator.java:145) ~[cluster-operator-0.6.0.jar:?]
  | at io.strimzi.operator.common.operator.resource.WorkaroundRbacOperator.lambda$doReconcile$0(WorkaroundRbacOperator.java:73) ~[cluster-operator-0.6.0.jar:?]
  | at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:273) ~[cluster-operator-0.6.0.jar:?]
  | at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[cluster-operator-0.6.0.jar:?]
  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [cluster-operator-0.6.0.jar:?]
  | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

[cameronbraid]

and it does this in a loop

[scholzj]

• Could increase the log level to DEBUG and share the full log? It should help to understand the problem. The log level for the cluster operator is configured in the environment variable in the deployment - so just change the deployment and modify it from INFO to DEBUG.
• Are you sure you installed all the RBAC files from the installation folder without any errors? Did you modify them to refer to the correct namespace?

[cameronbraid]

Its been a working cluster for a few months, then I noticed this happening.

Here are the logs

+ JAR=/cluster-operator-0.6.0.jar
--
  | + shift
  | + . /bin/dynamic_resources.sh
  | ++ get_heap_size
  | +++ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
  | ++ CONTAINER_MEMORY_IN_BYTES=536870912
  | ++ DEFAULT_MEMORY_CEILING=1152921504606846975
  | ++ '[' 536870912 -lt 1152921504606846975 ']'
  | ++ '[' -z ']'
  | ++ CONTAINER_HEAP_PERCENT=0.50
  | ++ CONTAINER_MEMORY_IN_MB=512
  | +++ echo '512 0.50'
  | +++ awk '{ printf "%d", $1 * $2 }'
  | ++ CONTAINER_HEAP_MAX=256
  | ++ echo 256
  | + MAX_HEAP=256
  | + '[' -n 256 ']'
  | + JAVA_OPTS='-Xms256m -Xmx256m '
  | + export MALLOC_ARENA_MAX=2
  | + MALLOC_ARENA_MAX=2
  | + JAVA_OPTS='-Xms256m -Xmx256m  -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
  | + JAVA_OPTS='-Xms256m -Xmx256m  -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps'
  | + exec java -Xms256m -Xmx256m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -jar /cluster-operator-0.6.0.jar -Xms256m -Xmx256m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
  | 2018-11-01 07:46:54 INFO  Main:60 - ClusterOperator is starting
  | 2018-11-01 07:46:54 DEBUG InternalLoggerFactory:71 - Using SLF4J as the default logging framework
  | 2018-11-01 07:46:54 DEBUG ResourceLeakDetector:81 - -Dio.netty.leakDetection.level: simple
  | 2018-11-01 07:46:54 DEBUG ResourceLeakDetector:81 - -Dio.netty.leakDetection.targetRecords: 4
  | 2018-11-01 07:46:54 DEBUG MultithreadEventLoopGroup:76 - -Dio.netty.eventLoopThreads: 96
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:76 - -Dio.netty.noUnsafe: false
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:76 - Java version: 8
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:71 - sun.misc.Unsafe.theUnsafe: available
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:71 - sun.misc.Unsafe.copyMemory: available
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:71 - java.nio.Buffer.address: available
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:71 - direct buffer constructor: available
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:76 - java.nio.Bits.unaligned: available, true
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:71 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
  | 2018-11-01 07:46:54 DEBUG PlatformDependent0:76 - java.nio.DirectByteBuffer.<init>(long, int): available
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:76 - sun.misc.Unsafe: available
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:76 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:76 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:76 - -Dio.netty.noPreferDirect: false
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:76 - -Dio.netty.maxDirectMemory: 257425408 bytes
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:76 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
  | 2018-11-01 07:46:54 DEBUG CleanerJava6:71 - java.nio.ByteBuffer.cleaner(): available
  | 2018-11-01 07:46:54 DEBUG NioEventLoop:76 - -Dio.netty.noKeySetOptimization: false
  | 2018-11-01 07:46:54 DEBUG NioEventLoop:76 - -Dio.netty.selectorAutoRebuildThreshold: 512
  | 2018-11-01 07:46:54 DEBUG PlatformDependent:71 - org.jctools-core.MpscChunkedArrayQueue: available
  | 2018-11-01 07:46:54 DEBUG DefaultDnsServerAddressStreamProvider:76 - Default DNS servers: [/168.1.123.248:53] (sun.net.dns.ResolverConfiguration)
  | 2018-11-01 07:46:54 DEBUG NetUtil:76 - -Djava.net.preferIPv4Stack: false
  | 2018-11-01 07:46:54 DEBUG NetUtil:76 - -Djava.net.preferIPv6Addresses: false
  | 2018-11-01 07:46:54 DEBUG NetUtil:86 - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
  | 2018-11-01 07:46:54 DEBUG NetUtil:81 - /proc/sys/net/core/somaxconn: 128
  | 2018-11-01 07:46:54 DEBUG Config:373 - Trying to configure client from Kubernetes config...
  | 2018-11-01 07:46:54 DEBUG Config:416 - Did not find Kubernetes config at: [/.kube/config]. Ignoring.
  | 2018-11-01 07:46:54 DEBUG Config:330 - Trying to configure client from service account...
  | 2018-11-01 07:46:54 DEBUG Config:334 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
  | 2018-11-01 07:46:54 DEBUG Config:342 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
  | 2018-11-01 07:46:54 DEBUG Config:423 - Trying to configure client namespace from Kubernetes service account namespace path...
  | 2018-11-01 07:46:54 DEBUG Config:428 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
  | 2018-11-01 07:46:55 DEBUG Main:165 - https://kubernetes.default.svc/oapi  returned 200. We are on OpenShift.
  | 2018-11-01T07:46:55.250+0000: [GC (Allocation Failure) [PSYoungGen: 65536K->10738K(76288K)] 65536K->19924K(251392K), 0.0509514 secs] [Times: user=0.05 sys=0.02, real=0.05 secs]
  | 2018-11-01T07:46:55.319+0000: [GC (Metadata GC Threshold) [PSYoungGen: 13097K->10743K(76288K)] 22283K->20481K(251392K), 0.1821137 secs] [Times: user=0.16 sys=0.02, real=0.18 secs]
  | 2018-11-01T07:46:55.501+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 10743K->0K(76288K)] [ParOldGen: 9738K->19697K(175104K)] 20481K->19697K(251392K), [Metaspace: 20797K->20797K(1069056K)], 1.0944713 secs] [Times: user=1.04 sys=0.01, real=1.09 secs]
  | 2018-11-01 07:46:56 INFO  Main:192 - Using config:
  | CRATE_PORT_4200_TCP_ADDR: 172.30.127.206
  | DIVOLTE_PORT_8290_TCP_PROTO: tcp
  | PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  | KUBERNETES_PORT_53_UDP_PROTO: udp
  | KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_REPLICATION: 9091
  | DIVOLTE_SERVICE_PORT_8290_TCP: 8290
  | WEBAPP_DISCOVERY_PORT_47500_TCP_ADDR: 172.30.148.31
  | KAFKA_REST_PROXY_PORT_8082_TCP_PORT: 8082
  | KAFKA_REST_PROXY_PORT_8082_TCP: tcp://172.30.171.161:8082
  | PHPMYADMIN_SERVICE_PORT: 8080
  | KAFKA_REST_PROXY_SERVICE_HOST: 172.30.171.161
  | CRATE_PORT_8080_TCP_PORT: 8080
  | BLOG_HTTP_SERVICE_PORT_HTTP: 80
  | STRIMZI_DEFAULT_KAFKA_IMAGE: strimzi/kafka:0.6.0
  | KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 172.30.1.80
  | SCHEMA_REGISTRY_UI_PORT_8000_TCP_PORT: 8000
  | CRATE_DISCOVERY_PORT_4300_TCP: tcp://172.30.217.78:4300
  | NITRO_SERVER_PORT_8080_TCP_PROTO: tcp
  | WEBDATA_HTTP_SERVICE_HOST: 172.30.15.94
  | CRATE_SERVICE_PORT_TRANSPORT: 4300
  | PHPMYADMIN_PORT: tcp://172.30.217.169:8080
  | BLOG_HTTP_SERVICE_HOST: 172.30.166.194
  | NITRO_SERVER_PORT_3001_TCP: tcp://172.30.241.12:3001
  | WEBAPP_PORT: tcp://172.30.156.227:8080
  | STRIMZI_DEFAULT_TLS_SIDECAR_KAFKA_IMAGE: strimzi/kafka-stunnel:0.6.0
  | XTRADB_BACKUP_RECEIVER_PORT_22_TCP_PROTO: tcp
  | ELASTICSEARCH_SERVICE_HOST: 172.30.126.124
  | XTRADB_BACKUP_RECEIVER_PORT: tcp://172.30.188.177:22
  | KEYCLOAK_SERVICE_PORT_HTTP_KEYCLOAK: 8080
  | ELASTICSEARCH_PORT_8080_TCP: tcp://172.30.126.124:8080
  | WEBAPP_SERVICE_HOST: 172.30.156.227
  | GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP_PROTO: tcp
  | KAFKA_EXPORTER_SERVICE_PORT_9308_TCP: 9308
  | XTRADB_SERVICE_PORT_3306_TCP: 3306
  | CRATE_DISCOVERY_SERVICE_PORT: 4200
  | STRIMZI_LOG_LEVEL: DEBUG
  | NITRO_SERVER_SERVICE_PORT_WEBAPP: 8080
  | TCP_ECHO_SERVICE_PORT: 3660
  | WEBAPP_SERVICE_PORT_CLUSTER_API: 8081
  | CRATE_PORT_4300_TCP: tcp://172.30.127.206:4300
  | REDIS_SENTINEL_SERVICE_HOST: 172.30.170.227
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP_PROTO: tcp
  | KAFKA_KAFKA_BOOTSTRAP_SERVICE_HOST: 172.30.55.27
  | KEYCLOAK_PORT_8080_TCP_ADDR: 172.30.157.33
  | ELASTICSEARCH_PORT_9200_TCP_PORT: 9200
  | CRATE_DISCOVERY_PORT_5432_TCP_PROTO: tcp
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 172.30.55.27
  | KAFKA_EXPORTER_PORT_9308_TCP_PROTO: tcp
  | CRATE_PORT_8080_TCP: tcp://172.30.127.206:8080
  | KUBERNETES_PORT: tcp://172.30.0.1:443
  | CRATE_DISCOVERY_SERVICE_PORT_CLUSTER: 4300
  | ELASTICSEARCH_DISCOVERY_PORT_9300_TCP_PROTO: tcp
  | ELASTICSEARCH_DISCOVERY_PORT_9200_TCP_PROTO: tcp
  | STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
  | CRATE_DISCOVERY_PORT_5432_TCP_ADDR: 172.30.217.78
  | CRATE_PORT_4300_TCP_ADDR: 172.30.127.206
  | WEBAPP_DISCOVERY_SERVICE_PORT_IGNITE_DISC: 47500
  | STRIMZI_DEFAULT_KAFKA_CONNECT_IMAGE: strimzi/kafka-connect:0.6.0
  | DIVOLTE_SERVICE_HOST: 172.30.31.135
  | PIWIK_PORT_80_TCP: tcp://172.30.226.160:80
  | TCP_ECHO_PORT_3660_TCP_ADDR: 172.30.111.226
  | KAFKA_EXPORTER_PORT_9308_TCP_PORT: 9308
  | ELASTICSEARCH_DISCOVERY_PORT_9200_TCP_ADDR: 172.30.212.20
  | CRATE_SERVICE_PORT: 8080
  | BLOG_HTTP_PORT: tcp://172.30.166.194:80
  | CRATE_DISCOVERY_PORT_4200_TCP_PORT: 4200
  | SMTP_RELAY_SERVICE_PORT: 25
  | NITRO_SERVER_SERVICE_HOST: 172.30.241.12
  | KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT_METRICS: 9404
  | SCHEMA_REGISTRY_UI_SERVICE_HOST: 172.30.223.175
  | PHPMYADMIN_PORT_80_TCP_ADDR: 172.30.217.169
  | CRATE_PORT: tcp://172.30.127.206:8080
  | CRATE_DISCOVERY_PORT_4300_TCP_PORT: 4300
  | CRATE_SERVICE_PORT_PROXY: 8080
  | KEYCLOAK_SERVICE_PORT: 8080
  | KUBERNETES_PORT_443_TCP_PROTO: tcp
  | WEBAPP_DISCOVERY_PORT_47100_TCP_PORT: 47100
  | WEBAPP_PORT_8080_TCP_ADDR: 172.30.156.227
  | KAFKA_CONNECT_UI_PORT_8000_TCP: tcp://172.30.74.76:8000
  | KUBERNETES_PORT_53_TCP_PORT: 53
  | PIWIK_PORT_80_TCP_PROTO: tcp
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 172.30.55.27
  | DIVOLTE_PORT: tcp://172.30.31.135:8290
  | KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_CLIENTS: 9092
  | KAFKA_CONNECT_UI_SERVICE_HOST: 172.30.74.76
  | MAILHOG_PORT_8080_TCP: tcp://172.30.173.196:8080
  | KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP_PORT: 8081
  | WEBAPP_DISCOVERY_PORT_47500_TCP: tcp://172.30.148.31:47500
  | WEBAPP_PORT_8081_TCP: tcp://172.30.156.227:8081
  | KAFKA_ZOOKEEPER_CLIENT_PORT: tcp://172.30.1.80:9404
  | GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP_ADDR: 172.30.9.52
  | MYSQL_PORT_3306_TCP: tcp://172.30.169.154:3306
  | MYSQL_PORT_3306_TCP_PROTO: tcp
  | NITRO_SERVER_PORT_3001_TCP_PORT: 3001
  | KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
  | SMTP_RELAY_PORT_25_TCP_ADDR: 172.30.124.75
  | KAFKA_REST_PROXY_PORT_8082_TCP_ADDR: 172.30.171.161
  | REDIS_SENTINEL_PORT_26379_TCP: tcp://172.30.170.227:26379
  | MAILHOG_PORT_8080_TCP_PORT: 8080
  | XTRADB_BACKUP_RECEIVER_PORT_22_TCP_PORT: 22
  | KAFKA_TOPICS_UI_PORT: tcp://172.30.150.150:8000
  | PHPMYADMIN_PORT_80_TCP_PROTO: tcp
  | KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_CLIENTSTLS: 9093
  | MAILHOG_SERVICE_HOST: 172.30.173.196
  | KAFKA_SCHEMA_REGISTRY_SERVICE_PORT: 8081
  | PIWIK_PORT_80_TCP_ADDR: 172.30.226.160
  | ELASTICSEARCH_PORT_8080_TCP_PORT: 8080
  | CRATE_DISCOVERY_PORT_4300_TCP_ADDR: 172.30.217.78
  | STRIMZI_DEFAULT_KAFKA_CONNECT_S2I_IMAGE: strimzi/kafka-connect-s2i:0.6.0
  | STRIMZI_VERSION: 0.6.0
  | KAFKA_REST_PROXY_PORT_8082_TCP_PROTO: tcp
  | XTRADB_BACKUP_RECEIVER_PORT_22_TCP: tcp://172.30.188.177:22
  | SMTP_RELAY_PORT: tcp://172.30.124.75:25
  | KEYCLOAK_PORT_8080_TCP_PORT: 8080
  | NITRO_SERVER_PORT: tcp://172.30.241.12:8080
  | WEBDATA_HTTP_PORT_8080_TCP_PORT: 8080
  | PIWIK_SERVICE_PORT: 80
  | CRATE_DISCOVERY_SERVICE_PORT_POSTGRES: 5432
  | REDIS_SENTINEL_PORT: tcp://172.30.170.227:26379
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_SERVICE_HOST: 172.30.132.9
  | KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP: tcp://172.30.91.229:8081
  | DIVOLTE_PORT_8290_TCP_PORT: 8290
  | XTRADB_BACKUP_RECEIVER_SERVICE_PORT_SSH: 22
  | XTRADB_BACKUP_RECEIVER_SERVICE_HOST: 172.30.188.177
  | KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT: 9404
  | PIWIK_PORT: tcp://172.30.226.160:80
  | WEBDATA_HTTP_PORT_8080_TCP: tcp://172.30.15.94:8080
  | WEBAPP_PORT_8080_TCP_PORT: 8080
  | BLOG_HTTP_PORT_80_TCP: tcp://172.30.166.194:80
  | SMTP_RELAY_SERVICE_PORT_SMTP: 25
  | KUBERNETES_PORT_53_TCP: tcp://172.30.0.1:53
  | KUBERNETES_PORT_53_TCP_PROTO: tcp
  | KAFKA_EXPORTER_SERVICE_PORT: 9308
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://172.30.55.27:9091
  | WEBAPP_DISCOVERY_SERVICE_PORT_IGNITE_COMM: 47100
  | WEBAPP_DISCOVERY_PORT_47100_TCP_ADDR: 172.30.148.31
  | XTRADB_SERVICE_PORT: 3306
  | CRATE_PORT_4200_TCP: tcp://172.30.127.206:4200
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP: tcp://172.30.55.27:9404
  | PHPMYADMIN_SERVICE_HOST: 172.30.217.169
  | MYSQL_SERVICE_HOST: 172.30.169.154
  | XTRADB_PORT_3306_TCP: tcp://172.30.57.121:3306
  | XTRADB_PORT: tcp://172.30.57.121:3306
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
  | WEBDATA_HTTP_SERVICE_PORT_HTTP: 8080
  | KUBERNETES_PORT_53_UDP: udp://172.30.0.1:53
  | NITRO_SERVER_PORT_3001_TCP_PROTO: tcp
  | KEYCLOAK_PORT_9990_TCP_PORT: 9990
  | PHPMYADMIN_PORT_8080_TCP_PORT: 8080
  | KUBERNETES_PORT_53_TCP_ADDR: 172.30.0.1
  | ELASTICSEARCH_DISCOVERY_SERVICE_HOST: 172.30.212.20
  | SCHEMA_REGISTRY_UI_PORT_8000_TCP_ADDR: 172.30.223.175
  | XTRADB_PORT_3306_TCP_PROTO: tcp
  | WEBAPP_DISCOVERY_SERVICE_PORT: 47500
  | KAFKA_CONNECT_UI_SERVICE_PORT_8000_TCP: 8000
  | SCHEMA_REGISTRY_UI_SERVICE_PORT_8000_TCP: 8000
  | PHPMYADMIN_PORT_80_TCP_PORT: 80
  | KUBERNETES_SERVICE_PORT: 443
  | CRATE_PORT_8080_TCP_PROTO: tcp
  | KAFKA_EXPORTER_PORT: tcp://172.30.44.108:9308
  | HOSTNAME: strimzi-cluster-operator-390472989-6f8sv
  | REDIS_SENTINEL_PORT_26379_TCP_PROTO: tcp
  | PHPMYADMIN_PORT_8080_TCP: tcp://172.30.217.169:8080
  | ELASTICSEARCH_SERVICE_PORT_HTTP: 8080
  | KAFKA_TOPICS_UI_SERVICE_PORT_8000_TCP: 8000
  | BLOG_HTTP_SERVICE_PORT: 80
  | CRATE_PORT_4300_TCP_PORT: 4300
  | KAFKA_REST_PROXY_SERVICE_PORT_8082_TCP: 8082
  | ELASTICSEARCH_DISCOVERY_PORT_9300_TCP_PORT: 9300
  | REDIS_SENTINEL_PORT_26379_TCP_ADDR: 172.30.170.227
  | HOME: /
  | KAFKA_EXPORTER_PORT_9308_TCP_ADDR: 172.30.44.108
  | MALLOC_ARENA_MAX: 2
  | STRIMZI_DEFAULT_ZOOKEEPER_IMAGE: strimzi/zookeeper:0.6.0
  | KUBERNETES_PORT_53_UDP_ADDR: 172.30.0.1
  | KAFKA_CONNECT_UI_PORT_8000_TCP_PORT: 8000
  | SMTP_RELAY_PORT_25_TCP_PORT: 25
  | ELASTICSEARCH_PORT_9300_TCP_ADDR: 172.30.126.124
  | KAFKA_CONNECT_UI_PORT_8000_TCP_PROTO: tcp
  | BLOG_HTTP_PORT_80_TCP_ADDR: 172.30.166.194
  | TCP_ECHO_SERVICE_PORT_3660_TCP: 3660
  | STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: strimzi/topic-operator:0.6.0
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
  | GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP_PORT: 1
  | CRATE_DISCOVERY_SERVICE_HOST: 172.30.217.78
  | MAILHOG_PORT_8080_TCP_PROTO: tcp
  | XTRADB_PORT_3306_TCP_PORT: 3306
  | KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP_ADDR: 172.30.91.229
  | MYSQL_PORT_3306_TCP_ADDR: 172.30.169.154
  | ELASTICSEARCH_PORT_9200_TCP_PROTO: tcp
  | ELASTICSEARCH_SERVICE_PORT_HTTP_PRIVATE: 9200
  | WEBDATA_HTTP_PORT_8080_TCP_ADDR: 172.30.15.94
  | WEBAPP_PORT_8080_TCP: tcp://172.30.156.227:8080
  | TCP_ECHO_SERVICE_HOST: 172.30.111.226
  | ELASTICSEARCH_PORT_9300_TCP_PROTO: tcp
  | DIVOLTE_PORT_8290_TCP_ADDR: 172.30.31.135
  | NITRO_SERVER_PORT_3001_TCP_ADDR: 172.30.241.12
  | KAFKA_REST_PROXY_PORT: tcp://172.30.171.161:8082
  | REDIS_SENTINEL_SERVICE_PORT: 26379
  | KAFKA_TOPICS_UI_SERVICE_PORT: 8000
  | KAFKA_SCHEMA_REGISTRY_PORT: tcp://172.30.91.229:8081
  | CRATE_DISCOVERY_PORT_5432_TCP: tcp://172.30.217.78:5432
  | PIWIK_PORT_80_TCP_PORT: 80
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 172.30.55.27
  | PWD: /
  | ELASTICSEARCH_DISCOVERY_PORT_9300_TCP: tcp://172.30.212.20:9300
  | GLUSTERFS_DYNAMIC_PLUGINS_SERVICE_PORT: 1
  | DIVOLTE_SERVICE_PORT: 8290
  | NITRO_SERVER_SERVICE_PORT_MANAGEMNENT: 3001
  | STRIMZI_NAMESPACE: drivenow-staging-x
  | ELASTICSEARCH_SERVICE_PORT: 8080
  | KEYCLOAK_SERVICE_HOST: 172.30.157.33
  | KAFKA_TOPICS_UI_PORT_8000_TCP_PORT: 8000
  | ELASTICSEARCH_PORT_8080_TCP_ADDR: 172.30.126.124
  | NITRO_SERVER_PORT_8080_TCP_ADDR: 172.30.241.12
  | KAFKA_CONNECT_UI_PORT: tcp://172.30.74.76:8000
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP_PORT: 9404
  | CRATE_DISCOVERY_PORT_5432_TCP_PORT: 5432
  | WEBAPP_SERVICE_PORT_WEBAPP: 8080
  | PHPMYADMIN_PORT_8080_TCP_PROTO: tcp
  | NITRO_SERVER_SERVICE_PORT: 8080
  | MAILHOG_PORT_8080_TCP_ADDR: 172.30.173.196
  | KUBERNETES_SERVICE_PORT_HTTPS: 443
  | SHLVL: 1
  | CRATE_DISCOVERY_PORT_4300_TCP_PROTO: tcp
  | WEBDATA_HTTP_SERVICE_PORT: 8080
  | KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
  | CRATE_DISCOVERY_PORT_4200_TCP_PROTO: tcp
  | DIVOLTE_PORT_8290_TCP: tcp://172.30.31.135:8290
  | JAVA_HOME: /usr/lib/jvm/java
  | WEBAPP_PORT_8080_TCP_PROTO: tcp
  | MAILHOG_SERVICE_PORT: 8080
  | WEBDATA_HTTP_PORT: tcp://172.30.15.94:8080
  | GLUSTERFS_DYNAMIC_PLUGINS_PORT: tcp://172.30.9.52:1
  | NITRO_SERVER_PORT_8080_TCP: tcp://172.30.241.12:8080
  | KAFKA_CONNECT_UI_SERVICE_PORT: 8000
  | KEYCLOAK_PORT_9990_TCP: tcp://172.30.157.33:9990
  | ELASTICSEARCH_PORT_9200_TCP_ADDR: 172.30.126.124
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_SERVICE_PORT: 1
  | KAFKA_KAFKA_BOOTSTRAP_PORT: tcp://172.30.55.27:9091
  | CRATE_PORT_4300_TCP_PROTO: tcp
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://172.30.55.27:9092
  | CRATE_PORT_4200_TCP_PROTO: tcp
  | KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://172.30.1.80:2181
  | MAILHOG_PORT: tcp://172.30.173.196:8080
  | XTRADB_BACKUP_RECEIVER_PORT_22_TCP_ADDR: 172.30.188.177
  | KUBERNETES_PORT_443_TCP_ADDR: 172.30.0.1
  | ELASTICSEARCH_DISCOVERY_PORT_9300_TCP_ADDR: 172.30.212.20
  | PHPMYADMIN_SERVICE_PORT_HTTP: 8080
  | REDIS_SENTINEL_PORT_26379_TCP_PORT: 26379
  | STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: strimzi/user-operator:0.6.0
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT: tcp://172.30.132.9:1
  | ELASTICSEARCH_SERVICE_PORT_TRANSPORT: 9300
  | CRATE_DISCOVERY_PORT: tcp://172.30.217.78:4200
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
  | ELASTICSEARCH_PORT_8080_TCP_PROTO: tcp
  | WEBAPP_PORT_8081_TCP_PORT: 8081
  | KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP_PORT: 9404
  | STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: strimzi/kafka-init:0.6.0
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
  | SCHEMA_REGISTRY_UI_SERVICE_PORT: 8000
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP_ADDR: 172.30.132.9
  | CRATE_SERVICE_HOST: 172.30.127.206
  | SMTP_RELAY_SERVICE_HOST: 172.30.124.75
  | KUBERNETES_SERVICE_PORT_DNS_TCP: 53
  | STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: strimzi/entity-operator-stunnel:0.6.0
  | WEBAPP_SERVICE_PORT: 8080
  | KEYCLOAK_PORT_9990_TCP_ADDR: 172.30.157.33
  | CRATE_PORT_8080_TCP_ADDR: 172.30.127.206
  | ELASTICSEARCH_DISCOVERY_SERVICE_PORT_TRANSPORT: 9300
  | KAFKA_TOPICS_UI_PORT_8000_TCP: tcp://172.30.150.150:8000
  | SMTP_RELAY_PORT_25_TCP: tcp://172.30.124.75:25
  | KAFKA_SCHEMA_REGISTRY_SERVICE_PORT_8081_TCP: 8081
  | SCHEMA_REGISTRY_UI_PORT: tcp://172.30.223.175:8000
  | MYSQL_PORT_3306_TCP_PORT: 3306
  | KAFKA_CONNECT_UI_PORT_8000_TCP_ADDR: 172.30.74.76
  | NITRO_SERVER_PORT_8080_TCP_PORT: 8080
  | CRATE_PORT_4200_TCP_PORT: 4200
  | KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP_ADDR: 172.30.1.80
  | WEBAPP_DISCOVERY_PORT_47500_TCP_PORT: 47500
  | ELASTICSEARCH_PORT_9300_TCP_PORT: 9300
  | PHPMYADMIN_PORT_8080_TCP_ADDR: 172.30.217.169
  | KAFKA_ZOOKEEPER_CLIENT_SERVICE_HOST: 172.30.1.80
  | KUBERNETES_PORT_53_UDP_PORT: 53
  | XTRADB_PORT_3306_TCP_ADDR: 172.30.57.121
  | SMTP_RELAY_PORT_25_TCP_PROTO: tcp
  | KAFKA_EXPORTER_PORT_9308_TCP: tcp://172.30.44.108:9308
  | WEBAPP_PORT_8081_TCP_ADDR: 172.30.156.227
  | KAFKA_TOPICS_UI_PORT_8000_TCP_PROTO: tcp
  | XTRADB_BACKUP_RECEIVER_SERVICE_PORT: 22
  | ELASTICSEARCH_PORT: tcp://172.30.126.124:8080
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP_ADDR: 172.30.55.27
  | KUBERNETES_PORT_443_TCP: tcp://172.30.0.1:443
  | ELASTICSEARCH_DISCOVERY_PORT: tcp://172.30.212.20:9300
  | ELASTICSEARCH_PORT_9300_TCP: tcp://172.30.126.124:9300
  | KEYCLOAK_PORT_8080_TCP_PROTO: tcp
  | SCHEMA_REGISTRY_UI_PORT_8000_TCP_PROTO: tcp
  | BLOG_HTTP_PORT_80_TCP_PROTO: tcp
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP: tcp://172.30.132.9:1
  | KEYCLOAK_PORT: tcp://172.30.157.33:8080
  | KAFKA_TOPICS_UI_PORT_8000_TCP_ADDR: 172.30.150.150
  | CRATE_DISCOVERY_PORT_4200_TCP_ADDR: 172.30.217.78
  | ELASTICSEARCH_DISCOVERY_SERVICE_PORT: 9300
  | SCHEMA_REGISTRY_UI_PORT_8000_TCP: tcp://172.30.223.175:8000
  | WEBAPP_DISCOVERY_PORT: tcp://172.30.148.31:47500
  | BLOG_HTTP_PORT_80_TCP_PORT: 80
  | TCP_ECHO_PORT_3660_TCP_PORT: 3660
  | ELASTICSEARCH_DISCOVERY_PORT_9200_TCP_PORT: 9200
  | KEYCLOAK_SERVICE_PORT_HTTP_ADMIN: 9990
  | ELASTICSEARCH_PORT_9200_TCP: tcp://172.30.126.124:9200
  | KAFKA_TOPICS_UI_SERVICE_HOST: 172.30.150.150
  | KEYCLOAK_PORT_9990_TCP_PROTO: tcp
  | PIWIK_SERVICE_HOST: 172.30.226.160
  | TCP_ECHO_PORT_3660_TCP_PROTO: tcp
  | CRATE_DISCOVERY_PORT_4200_TCP: tcp://172.30.217.78:4200
  | KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_METRICS: 9404
  | ELASTICSEARCH_DISCOVERY_SERVICE_PORT_TRANSPORT9200: 9200
  | KAFKA_SCHEMA_REGISTRY_SERVICE_HOST: 172.30.91.229
  | WEBAPP_DISCOVERY_PORT_47500_TCP_PROTO: tcp
  | KAFKA_REST_PROXY_SERVICE_PORT: 8082
  | KUBERNETES_SERVICE_HOST: 172.30.0.1
  | CRATE_SERVICE_PORT_HTTP: 4200
  | MYSQL_PORT: tcp://172.30.169.154:3306
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
  | GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP: tcp://172.30.9.52:1
  | TCP_ECHO_PORT: tcp://172.30.111.226:3660
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
  | KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT_CLIENTS: 2181
  | KUBERNETES_SERVICE_PORT_DNS: 53
  | KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP: tcp://172.30.1.80:9404
  | CRATE_DISCOVERY_SERVICE_PORT_CRATE_WEB: 4200
  | STRIMZI_OPERATION_TIMEOUT_MS: 300000
  | KAFKA_EXPORTER_SERVICE_HOST: 172.30.44.108
  | XTRADB_SERVICE_HOST: 172.30.57.121
  | PHPMYADMIN_SERVICE_PORT_HTTP_PRIVATE: 80
  | KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP_PROTO: tcp
  | ELASTICSEARCH_DISCOVERY_PORT_9200_TCP: tcp://172.30.212.20:9200
  | WEBDATA_HTTP_PORT_8080_TCP_PROTO: tcp
  | PHPMYADMIN_PORT_80_TCP: tcp://172.30.217.169:80
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://172.30.55.27:9093
  | WEBAPP_DISCOVERY_PORT_47100_TCP: tcp://172.30.148.31:47100
  | KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
  | GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP_PORT: 1
  | WEBAPP_DISCOVERY_SERVICE_HOST: 172.30.148.31
  | KEYCLOAK_PORT_8080_TCP: tcp://172.30.157.33:8080
  | KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP_PROTO: tcp
  | TCP_ECHO_PORT_3660_TCP: tcp://172.30.111.226:3660
  | KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP_PROTO: tcp
  | MAILHOG_SERVICE_PORT_HTTP: 8080
  | GLUSTERFS_DYNAMIC_PLUGINS_SERVICE_HOST: 172.30.9.52
  | WEBAPP_PORT_8081_TCP_PROTO: tcp
  | KUBERNETES_PORT_443_TCP_PORT: 443
  | MYSQL_SERVICE_PORT: 3306
  | WEBAPP_DISCOVERY_PORT_47100_TCP_PROTO: tcp
  | STRIMZI_DEFAULT_TLS_SIDECAR_ZOOKEEPER_IMAGE: strimzi/zookeeper-stunnel:0.6.0
  | 2018-11-01 07:46:56 INFO  ClusterOperator:55 - Creating ClusterOperator for namespace drivenow-staging-x
  | 2018-11-01 07:46:56 INFO  ClusterOperator:82 - Starting ClusterOperator for namespace drivenow-staging-x
  | 2018-11-01 07:46:56 DEBUG WatchConnectionManager:101 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@25588c29
  | 2018-11-01 07:46:57 DEBUG WatchConnectionManager:151 - WebSocket successfully opened
  | 2018-11-01 07:46:57 INFO  ClusterOperator:89 - Started operator for Kafka kind
  | 2018-11-01 07:46:57 DEBUG WatchConnectionManager:101 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@9f025af
  | 2018-11-01 07:46:57 DEBUG WatchConnectionManager:151 - WebSocket successfully opened
  | 2018-11-01 07:46:57 INFO  ClusterOperator:94 - Started operator for KafkaConnect kind
  | 2018-11-01 07:46:57 DEBUG WatchConnectionManager:101 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@27ce09
  | 2018-11-01 07:46:57 DEBUG WatchConnectionManager:151 - WebSocket successfully opened
  | 2018-11-01 07:46:57 INFO  ClusterOperator:104 - Started operator for KafkaConnectS2I kind
  | 2018-11-01 07:46:57 INFO  ClusterOperator:107 - Setting up periodical reconciliation for namespace drivenow-staging-x
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.numHeapArenas: 2
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.numDirectArenas: 2
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.pageSize: 8192
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.maxOrder: 11
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.chunkSize: 16777216
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.tinyCacheSize: 512
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.smallCacheSize: 256
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.normalCacheSize: 64
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.cacheTrimInterval: 8192
  | 2018-11-01 07:46:57 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.useCacheForAllThreads: true
  | 2018-11-01 07:46:57 DEBUG InternalThreadLocalMap:76 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
  | 2018-11-01 07:46:57 DEBUG InternalThreadLocalMap:76 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
  | 2018-11-01 07:46:57 INFO  AbstractAssemblyOperator:352 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): Kafka kafka in namespace drivenow-staging-x was ADDED
  | 2018-11-01 07:46:57 DEBUG AbstractAssemblyOperator:220 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): Lock lock::drivenow-staging-x::Kafka::kafka acquired
  | 2018-11-01 07:46:57 DEBUG DefaultChannelId:76 - -Dio.netty.processId: 1 (auto-detected)
  | 2018-11-01 07:46:57 DEBUG DefaultChannelId:76 - -Dio.netty.machineId: 0a:58:0a:ff:fe:82:01:c3 (auto-detected)
  | 2018-11-01 07:46:57 INFO  AbstractAssemblyOperator:228 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): Assembly kafka should be created or updated
  | 2018-11-01 07:46:57 DEBUG ByteBufUtil:76 - -Dio.netty.allocator.type: pooled
  | 2018-11-01 07:46:57 DEBUG ByteBufUtil:76 - -Dio.netty.threadLocalDirectBufferSize: 65536
  | 2018-11-01 07:46:57 DEBUG ByteBufUtil:76 - -Dio.netty.maxThreadLocalCharBufferSize: 16384
  | 2018-11-01 07:46:57 DEBUG AbstractAssemblyOperator:164 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): The cluster CA kafka-cluster-ca already exists
  | 2018-11-01 07:46:57 DEBUG SecretOperator:87 - Secret drivenow-staging-x/kafka-cluster-ca already exists, patching it
  | 2018-11-01 07:46:57 INFO  ClusterOperator:160 - ClusterOperator is now ready (health server listening on 8080)
  | 2018-11-01 07:46:57 INFO  Main:114 - Cluster Operator verticle started in namespace drivenow-staging-x
  | 2018-11-01 07:46:57 DEBUG SecretOperator:130 - Secret kafka-cluster-ca in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:46:57 DEBUG KafkaAssemblyOperator:517 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): create/update zookeeper kafka
  | 2018-11-01 07:46:58 DEBUG AbstractModel:188 - Generating certificates
  | 2018-11-01 07:46:58 DEBUG AbstractModel:206 - Cluster communication certificates
  | 2018-11-01 07:46:58 DEBUG AbstractModel:1038 - kafka-zookeeper-0 already exists
  | 2018-11-01 07:46:58 DEBUG AbstractModel:1038 - kafka-zookeeper-1 already exists
  | 2018-11-01 07:46:58 DEBUG AbstractModel:1038 - kafka-zookeeper-2 already exists
  | 2018-11-01 07:46:58 DEBUG AbstractModel:215 - End generating certificates
  | 2018-11-01 07:46:58 DEBUG ServiceOperator:87 - Service drivenow-staging-x/kafka-zookeeper-client already exists, patching it
  | 2018-11-01 07:46:58 DEBUG ServiceOperator:130 - Service kafka-zookeeper-client in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:46:58 DEBUG ServiceOperator:87 - Service drivenow-staging-x/kafka-zookeeper-nodes already exists, patching it
  | 2018-11-01 07:46:58 DEBUG ServiceOperator:130 - Service kafka-zookeeper-nodes in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:46:58 DEBUG ConfigMapOperator:87 - ConfigMap drivenow-staging-x/kafka-zookeeper-config already exists, patching it
  | 2018-11-01 07:46:58 DEBUG ConfigMapOperator:47 - ConfigMap kafka-zookeeper-config in namespace drivenow-staging-x has not been patched because resources are equal
  | 2018-11-01 07:46:58 DEBUG SecretOperator:87 - Secret drivenow-staging-x/kafka-zookeeper-nodes already exists, patching it
  | 2018-11-01 07:46:58 DEBUG SecretOperator:130 - Secret kafka-zookeeper-nodes in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:46:58 DEBUG ZookeeperSetOperator:87 - StatefulSet drivenow-staging-x/kafka-zookeeper already exists, patching it
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/revisionHistoryLimit"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/metadata/annotations"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/imagePullPolicy"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/successThreshold"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/successThreshold"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/resources"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePath"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePolicy"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/imagePullPolicy"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/resources"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePath"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePolicy"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/dnsPolicy"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/restartPolicy"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/schedulerName"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/securityContext"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/terminationGracePeriodSeconds"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/0/configMap/defaultMode"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/1/secret/defaultMode"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/status"}
  | 2018-11-01 07:46:59 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-zookeeper ignoring diff {"op":"remove","path":"/status"}
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:242 - Patching StatefulSet drivenow-staging-x/kafka-zookeeper
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:245 - Patched StatefulSet drivenow-staging-x/kafka-zookeeper
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:77 - Considering rolling update of drivenow-staging-x/kafka-zookeeper
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-staging-x/kafka-zookeeper: pod kafka-zookeeper-0 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-staging-x/kafka-zookeeper: pod kafka-zookeeper-0 has operator.strimzi.io/statefulset-generation=0; no need to roll
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-staging-x/kafka-zookeeper: pod kafka-zookeeper-1 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-staging-x/kafka-zookeeper: pod kafka-zookeeper-1 has operator.strimzi.io/statefulset-generation=0; no need to roll
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-staging-x/kafka-zookeeper: pod kafka-zookeeper-2 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
  | 2018-11-01 07:46:59 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-staging-x/kafka-zookeeper: pod kafka-zookeeper-2 has operator.strimzi.io/statefulset-generation=0; no need to roll
  | 2018-11-01 07:46:59 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-zookeeper-client in namespace drivenow-staging-x to get ready
  | 2018-11-01 07:46:59 DEBUG EndpointOperator:88 - Endpoints kafka-zookeeper-client in namespace drivenow-staging-x is ready
  | 2018-11-01 07:46:59 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-zookeeper-nodes in namespace drivenow-staging-x to get ready
  | 2018-11-01 07:46:59 DEBUG EndpointOperator:88 - Endpoints kafka-zookeeper-nodes in namespace drivenow-staging-x is ready
  | 2018-11-01 07:46:59 DEBUG KafkaAssemblyOperator:420 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): create/update kafka kafka
  | 2018-11-01 07:46:59 DEBUG AbstractModel:248 - Generating certificates
  | 2018-11-01 07:46:59 DEBUG AbstractModel:280 - Clients CA already exists
  | 2018-11-01 07:46:59 DEBUG AbstractModel:292 - Internal communication certificates
  | 2018-11-01 07:46:59 DEBUG AbstractModel:1038 - kafka-kafka-0 already exists
  | 2018-11-01 07:46:59 DEBUG AbstractModel:1038 - kafka-kafka-1 already exists
  | 2018-11-01 07:46:59 DEBUG AbstractModel:1038 - kafka-kafka-2 already exists
  | 2018-11-01 07:46:59 DEBUG AbstractModel:302 - End generating certificates
  | 2018-11-01T07:46:59.511+0000: [GC (Allocation Failure) [PSYoungGen: 65536K->10721K(76288K)] 85233K->33589K(251392K), 0.1894586 secs] [Times: user=0.17 sys=0.00, real=0.19 secs]
  | 2018-11-01 07:46:59 DEBUG ServiceAccountOperator:87 - ServiceAccount drivenow-staging-x/kafka-kafka already exists, patching it
  | 2018-11-01 07:46:59 DEBUG ClusterRoleBindingOperator:107 - Making GET request Request{method=GET, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/strimzi-kafka-kafka-init , tag=null}
  | 2018-11-01 07:46:59 DEBUG ClusterRoleBindingOperator:110 - Got GET response Response{protocol=http/1.1, code=404, message=Not Found, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/strimzi-kafka-kafka-init }
  | 2018-11-01 07:46:59 DEBUG ClusterRoleBindingOperator:76 - No-op (deletion requested, but does not exist)
  | 2018-11-01 07:46:59 DEBUG ServiceOperator:87 - Service drivenow-staging-x/kafka-kafka-bootstrap already exists, patching it
  | 2018-11-01 07:46:59 DEBUG ServiceOperator:130 - Service kafka-kafka-bootstrap in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:46:59 DEBUG ServiceOperator:87 - Service drivenow-staging-x/kafka-kafka-brokers already exists, patching it
  | 2018-11-01 07:46:59 DEBUG ServiceOperator:130 - Service kafka-kafka-brokers in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:47:00 DEBUG ConfigMapOperator:87 - ConfigMap drivenow-staging-x/kafka-kafka-config already exists, patching it
  | 2018-11-01 07:47:00 DEBUG ConfigMapOperator:47 - ConfigMap kafka-kafka-config in namespace drivenow-staging-x has not been patched because resources are equal
  | 2018-11-01 07:47:00 DEBUG SecretOperator:87 - Secret drivenow-staging-x/kafka-clients-ca already exists, patching it
  | 2018-11-01 07:47:00 DEBUG SecretOperator:130 - Secret kafka-clients-ca in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:47:00 DEBUG SecretOperator:87 - Secret drivenow-staging-x/kafka-clients-ca-cert already exists, patching it
  | 2018-11-01 07:47:00 DEBUG SecretOperator:130 - Secret kafka-clients-ca-cert in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:47:00 DEBUG SecretOperator:87 - Secret drivenow-staging-x/kafka-cluster-ca-cert already exists, patching it
  | 2018-11-01 07:47:00 DEBUG SecretOperator:130 - Secret kafka-cluster-ca-cert in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:47:00 DEBUG SecretOperator:87 - Secret drivenow-staging-x/kafka-kafka-brokers already exists, patching it
  | 2018-11-01 07:47:00 DEBUG SecretOperator:130 - Secret kafka-kafka-brokers in namespace drivenow-staging-x has been patched
  | 2018-11-01 07:47:00 DEBUG KafkaSetOperator:87 - StatefulSet drivenow-staging-x/kafka-kafka already exists, patching it
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/revisionHistoryLimit"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/metadata/annotations"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/imagePullPolicy"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/successThreshold"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/successThreshold"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/resources"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePath"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePolicy"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/imagePullPolicy"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/resources"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePath"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePolicy"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/dnsPolicy"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/restartPolicy"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/schedulerName"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/securityContext"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/serviceAccount"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/terminationGracePeriodSeconds"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/0/secret/defaultMode"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/1/secret/defaultMode"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/2/configMap/defaultMode"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/status"}
  | 2018-11-01 07:47:00 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-staging-x/kafka-kafka ignoring diff {"op":"remove","path":"/status"}
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:242 - Patching StatefulSet drivenow-staging-x/kafka-kafka
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:245 - Patched StatefulSet drivenow-staging-x/kafka-kafka
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:77 - Considering rolling update of drivenow-staging-x/kafka-kafka
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-staging-x/kafka-kafka: pod kafka-kafka-0 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-staging-x/kafka-kafka: pod kafka-kafka-0 has operator.strimzi.io/statefulset-generation=0; no need to roll
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-staging-x/kafka-kafka: pod kafka-kafka-1 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-staging-x/kafka-kafka: pod kafka-kafka-1 has operator.strimzi.io/statefulset-generation=0; no need to roll
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-staging-x/kafka-kafka: pod kafka-kafka-2 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
  | 2018-11-01 07:47:00 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-staging-x/kafka-kafka: pod kafka-kafka-2 has operator.strimzi.io/statefulset-generation=0; no need to roll
  | 2018-11-01 07:47:00 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-kafka-bootstrap in namespace drivenow-staging-x to get ready
  | 2018-11-01 07:47:00 DEBUG EndpointOperator:88 - Endpoints kafka-kafka-bootstrap in namespace drivenow-staging-x is ready
  | 2018-11-01 07:47:00 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-kafka-brokers in namespace drivenow-staging-x to get ready
  | 2018-11-01 07:47:00 DEBUG EndpointOperator:88 - Endpoints kafka-kafka-brokers in namespace drivenow-staging-x is ready
  | 2018-11-01 07:47:00 DEBUG KafkaAssemblyOperator:600 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): create/update topic operator kafka
  | 2018-11-01 07:47:00 DEBUG ServiceAccountOperator:96 - ServiceAccount drivenow-staging-x/kafka-topic-operator does not exist, noop
  | 2018-11-01 07:47:00 DEBUG RoleBindingOperator:107 - Making GET request Request{method=GET, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-staging-x/rolebindings/strimzi-topic-operator-role-binding , tag=null}
  | 2018-11-01 07:47:00 DEBUG RoleBindingOperator:110 - Got GET response Response{protocol=http/1.1, code=404, message=Not Found, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-staging-x/rolebindings/strimzi-topic-operator-role-binding }
  | 2018-11-01 07:47:00 DEBUG RoleBindingOperator:76 - No-op (deletion requested, but does not exist)
  | 2018-11-01 07:47:00 DEBUG ConfigMapOperator:96 - ConfigMap drivenow-staging-x/kafka-topic-operator-config does not exist, noop
  | 2018-11-01 07:47:00 DEBUG DeploymentOperator:96 - Deployment drivenow-staging-x/kafka-topic-operator does not exist, noop
  | 2018-11-01 07:47:00 DEBUG SecretOperator:96 - Secret drivenow-staging-x/kafka-topic-operator-certs does not exist, noop
  | 2018-11-01 07:47:00 DEBUG KafkaAssemblyOperator:687 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): create/update entity operator kafka
  | 2018-11-01 07:47:00 DEBUG AbstractModel:164 - Generating certificates
  | 2018-11-01 07:47:00 DEBUG AbstractModel:192 - Entity Operator certificate already exists
  | 2018-11-01 07:47:00 DEBUG AbstractAssemblyOperator:239 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): Lock lock::drivenow-staging-x::Kafka::kafka released
  | 2018-11-01 07:47:00 ERROR AbstractAssemblyOperator:244 - Reconciliation #0(watch) Kafka(drivenow-staging-x/kafka): createOrUpdate failed
  | java.lang.NullPointerException: null
  | at java.util.Base64$Decoder.decode(Base64.java:549) ~[?:1.8.0_181]
  | at io.strimzi.operator.cluster.model.AbstractModel.decodeFromSecret(AbstractModel.java:1015) ~[cluster-operator-0.6.0.jar:?]
  | at io.strimzi.operator.cluster.model.EntityOperator.generateCertificates(EntityOperator.java:195) ~[cluster-operator-0.6.0.jar:?]
  | at io.strimzi.operator.cluster.model.EntityOperator.fromCrd(EntityOperator.java:151) ~[cluster-operator-0.6.0.jar:?]
  | at io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperator.lambda$getEntityOperatorDescription$43(KafkaAssemblyOperator.java:645) ~[cluster-operator-0.6.0.jar:?]
  | at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:273) ~[cluster-operator-0.6.0.jar:?]
  | at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[cluster-operator-0.6.0.jar:?]
  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [cluster-operator-0.6.0.jar:?]
  | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  | 2018-11-01 07:47:21 DEBUG Recycler:76 - -Dio.netty.recycler.maxCapacityPerThread: 32768
  | 2018-11-01 07:47:21 DEBUG Recycler:76 - -Dio.netty.recycler.maxSharedCapacityFactor: 2
  | 2018-11-01 07:47:21 DEBUG Recycler:76 - -Dio.netty.recycler.linkCapacity: 16
  | 2018-11-01 07:47:21 DEBUG Recycler:76 - -Dio.netty.recycler.ratio: 8
  | 2018-11-01 07:47:21 DEBUG AbstractByteBuf:81 - -Dio.netty.buffer.bytebuf.checkAccessible: true
  | 2018-11-01 07:47:21 DEBUG ResourceLeakDetectorFactory:76 - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@38ed3a50

[cameronbraid]

Sorry, I have issues in two projects and I just pasted the logs from the wrong project, let me do it in the other project.

[cameronbraid]

+ JAR=/cluster-operator-0.6.0.jar
+ shift
+ . /bin/dynamic_resources.sh
++ get_heap_size
+++ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
++ CONTAINER_MEMORY_IN_BYTES=536870912
++ DEFAULT_MEMORY_CEILING=1152921504606846975
++ '[' 536870912 -lt 1152921504606846975 ']'
++ '[' -z ']'
++ CONTAINER_HEAP_PERCENT=0.50
++ CONTAINER_MEMORY_IN_MB=512
+++ echo '512 0.50'
+++ awk '{ printf "%d", $1 * $2 }'
++ CONTAINER_HEAP_MAX=256
++ echo 256
+ MAX_HEAP=256
+ '[' -n 256 ']'
+ JAVA_OPTS='-Xms256m -Xmx256m '
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS='-Xms256m -Xmx256m  -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
+ JAVA_OPTS='-Xms256m -Xmx256m  -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps'
+ exec java -Xms256m -Xmx256m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -jar /cluster-operator-0.6.0.jar -Xms256m -Xmx256m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
2018-11-01 07:51:07 INFO  Main:60 - ClusterOperator is starting
2018-11-01 07:51:07 DEBUG InternalLoggerFactory:71 - Using SLF4J as the default logging framework
2018-11-01 07:51:07 DEBUG ResourceLeakDetector:81 - -Dio.netty.leakDetection.level: simple
2018-11-01 07:51:07 DEBUG ResourceLeakDetector:81 - -Dio.netty.leakDetection.targetRecords: 4
2018-11-01 07:51:07 DEBUG MultithreadEventLoopGroup:76 - -Dio.netty.eventLoopThreads: 96
2018-11-01 07:51:07 DEBUG PlatformDependent0:76 - -Dio.netty.noUnsafe: false
2018-11-01 07:51:07 DEBUG PlatformDependent0:76 - Java version: 8
2018-11-01 07:51:07 DEBUG PlatformDependent0:71 - sun.misc.Unsafe.theUnsafe: available
2018-11-01 07:51:07 DEBUG PlatformDependent0:71 - sun.misc.Unsafe.copyMemory: available
2018-11-01 07:51:07 DEBUG PlatformDependent0:71 - java.nio.Buffer.address: available
2018-11-01 07:51:07 DEBUG PlatformDependent0:71 - direct buffer constructor: available
2018-11-01 07:51:07 DEBUG PlatformDependent0:76 - java.nio.Bits.unaligned: available, true
2018-11-01 07:51:07 DEBUG PlatformDependent0:71 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2018-11-01 07:51:07 DEBUG PlatformDependent0:76 - java.nio.DirectByteBuffer.<init>(long, int): available
2018-11-01 07:51:07 DEBUG PlatformDependent:76 - sun.misc.Unsafe: available
2018-11-01 07:51:07 DEBUG PlatformDependent:76 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2018-11-01 07:51:07 DEBUG PlatformDependent:76 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2018-11-01 07:51:07 DEBUG PlatformDependent:76 - -Dio.netty.noPreferDirect: false
2018-11-01 07:51:07 DEBUG PlatformDependent:76 - -Dio.netty.maxDirectMemory: 257425408 bytes
2018-11-01 07:51:07 DEBUG PlatformDependent:76 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2018-11-01 07:51:07 DEBUG CleanerJava6:71 - java.nio.ByteBuffer.cleaner(): available
2018-11-01 07:51:07 DEBUG NioEventLoop:76 - -Dio.netty.noKeySetOptimization: false
2018-11-01 07:51:07 DEBUG NioEventLoop:76 - -Dio.netty.selectorAutoRebuildThreshold: 512
2018-11-01 07:51:07 DEBUG PlatformDependent:71 - org.jctools-core.MpscChunkedArrayQueue: available
2018-11-01 07:51:07 DEBUG DefaultDnsServerAddressStreamProvider:76 - Default DNS servers: [/168.1.123.248:53] (sun.net.dns.ResolverConfiguration)
2018-11-01 07:51:07 DEBUG NetUtil:76 - -Djava.net.preferIPv4Stack: false
2018-11-01 07:51:07 DEBUG NetUtil:76 - -Djava.net.preferIPv6Addresses: false
2018-11-01 07:51:07 DEBUG NetUtil:86 - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
2018-11-01 07:51:07 DEBUG NetUtil:81 - /proc/sys/net/core/somaxconn: 128
2018-11-01 07:51:07 DEBUG Config:373 - Trying to configure client from Kubernetes config...
2018-11-01 07:51:07 DEBUG Config:416 - Did not find Kubernetes config at: [/.kube/config]. Ignoring.
2018-11-01 07:51:07 DEBUG Config:330 - Trying to configure client from service account...
2018-11-01 07:51:07 DEBUG Config:334 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
2018-11-01 07:51:07 DEBUG Config:342 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2018-11-01 07:51:07 DEBUG Config:423 - Trying to configure client namespace from Kubernetes service account namespace path...
2018-11-01 07:51:07 DEBUG Config:428 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2018-11-01 07:51:08 DEBUG Main:165 - https://kubernetes.default.svc/oapi returned 200. We are on OpenShift.
2018-11-01T07:51:08.411+0000: [GC (Allocation Failure) [PSYoungGen: 65536K->10721K(76288K)] 65536K->20065K(251392K), 0.1898470 secs] [Times: user=0.19 sys=0.02, real=0.19 secs] 
2018-11-01T07:51:08.621+0000: [GC (Metadata GC Threshold) [PSYoungGen: 13673K->10737K(76288K)] 23017K->20728K(251392K), 0.2803738 secs] [Times: user=0.24 sys=0.03, real=0.28 secs] 
2018-11-01T07:51:08.901+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 10737K->0K(76288K)] [ParOldGen: 9990K->14649K(175104K)] 20728K->14649K(251392K), [Metaspace: 20807K->20807K(1069056K)], 0.3990736 secs] [Times: user=0.36 sys=0.01, real=0.40 secs] 
2018-11-01 07:51:09 INFO  Main:192 - Using config:
    PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    KUBERNETES_PORT_53_UDP_PROTO: udp
    HTTP_REDIRECT_DRIVENOW_CO_NZ_PORT_8080_TCP_ADDR: 172.30.125.15
    DIVOLTE_SERVICE_PORT_8290_TCP: 8290
    KAFKA_REST_PROXY_PORT_8082_TCP_PORT: 8082
    CRATE_EXPORT_PORT_8080_TCP_PROTO: tcp
    KAFKA_REST_PROXY_PORT_8082_TCP: tcp://172.30.248.179:8082
    KAFKA_REST_PROXY_SERVICE_HOST: 172.30.248.179
    OFFLINE_SERVICE_HOST: 172.30.68.82
    HTTP_REDIRECT_DRIVENOW_COM_AU_PORT: tcp://172.30.182.239:8080
    BLOG_HTTP_SERVICE_PORT_HTTP: 80
    OSM_CACHE_PORT_80_TCP: tcp://172.30.125.26:80
    CRATE_DISCOVERY_PORT_4300_TCP: tcp://172.30.184.166:4300
    NITRO_SERVER_PORT_8080_TCP_PROTO: tcp
    CRATE_SERVICE_PORT_TRANSPORT: 4300
    PHPMYADMIN_PORT: tcp://172.30.59.23:8080
    BLOG_HTTP_SERVICE_HOST: 172.30.42.85
    NITRO_SERVER_PORT_3001_TCP: tcp://172.30.239.225:3001
    KEYCLOAK_SERVICE_PORT_HTTP_KEYCLOAK: 8080
    CF_TIMEOUT_PORT_8080_TCP_PORT: 8080
    CRATE_EXPORT_PORT_8081_TCP: tcp://172.30.62.157:8081
    TILESERVER_GL_PORT: tcp://172.30.43.195:80
    CRATE_EXPORT_SERVICE_HOST: 172.30.62.157
    NITRO_SERVER_SERVICE_PORT_WEBAPP: 8080
    WEBAPP_SERVICE_PORT_CLUSTER_API: 8081
    CRATE_PORT_4300_TCP: tcp://172.30.241.20:4300
    REDIS_SENTINEL_SERVICE_HOST: 172.30.46.154
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP_PROTO: tcp
    SIXPACK_REDIS_PORT_6379_TCP_PROTO: tcp
    KAFKA_KAFKA_BOOTSTRAP_SERVICE_HOST: 172.30.225.203
    GLUSTERFS_DYNAMIC_WEBDATA_PORT_1_TCP_ADDR: 172.30.119.83
    SIXPACK_WEB_PORT_8080_TCP_PORT: 8080
    CRATE_PORT_8080_TCP: tcp://172.30.241.20:8080
    CRATE_DISCOVERY_SERVICE_PORT_CLUSTER: 4300
    XERO_SERVICE_PORT_HTTP: 8180
    WEBAPP_MAINTENANCE_PORT: tcp://172.30.243.52:8080
    ELASTICSEARCH_DISCOVERY_PORT_9300_TCP_PROTO: tcp
    ELASTICSEARCH_DISCOVERY_PORT_9200_TCP_PROTO: tcp
    OSM_CACHE_SERVICE_HOST: 172.30.125.26
    STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
    TILESERVER_GL_SERVICE_HOST: 172.30.43.195
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_PORT_1_TCP_PORT: 1
    CRATE_PORT_4300_TCP_ADDR: 172.30.241.20
    STRIMZI_DEFAULT_KAFKA_CONNECT_IMAGE: strimzi/kafka-connect:0.6.0
    GLUSTERFS_DYNAMIC_WEBDATA_SERVICE_HOST: 172.30.119.83
    CRATE_SERVICE_PORT: 8080
    CRATE_DISCOVERY_PORT_4200_TCP_PORT: 4200
    TOPICS_UI_PORT_8090_TCP: tcp://172.30.122.9:8090
    SIXPACK_API_PORT_5000_TCP_PORT: 5000
    GLUSTERFS_CLUSTER_PORT_1_TCP_PORT: 1
    FILEBROWSER_PORT_8080_TCP_ADDR: 172.30.73.131
    KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT_METRICS: 9404
    REDIRECT_CAMPERVANHIRENOW_SERVICE_PORT_HTTP: 8080
    CRATE_PORT: tcp://172.30.241.20:8080
    XERO_PORT_8180_TCP_PORT: 8180
    THUMBOR_PORT_8000_TCP_PORT: 8000
    KEYCLOAK_SERVICE_PORT: 8080
    TOPICS_UI_SERVICE_HOST: 172.30.122.9
    KUBERNETES_PORT_443_TCP_PROTO: tcp
    SIXPACK_WEB_CT_PORT_8080_TCP: tcp://172.30.99.193:8080
    SIXPACK_WEB_SERVICE_PORT_SIXPACK_WEB: 8080
    WEBAPP_PORT_8080_TCP_ADDR: 172.30.205.114
    TILESERVER_GL_PORT_80_TCP: tcp://172.30.43.195:80
    KUBERNETES_PORT_53_TCP_PORT: 53
    PIWIK_PORT_80_TCP_PROTO: tcp
    KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 172.30.225.203
    DIVOLTE_PORT: tcp://172.30.42.87:8290
    EMAILQ_SERVICE_HOST: 172.30.246.123
    HTTP_REDIRECT_DRIVENOW_CO_NZ_PORT_8080_TCP: tcp://172.30.125.15:8080
    KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_CLIENTS: 9092
    KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP_PORT: 8081
    WEBAPP_MAINTENANCE_PORT_8081_TCP_PORT: 8081
    WEBAPP_PORT_8081_TCP: tcp://172.30.205.114:8081
    GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP_ADDR: 172.30.126.79
    GLUSTERFS_CLUSTER_SERVICE_PORT: 1
    TILESERVER_GL_PORT_80_TCP_ADDR: 172.30.43.195
    LLP_CONVERTER_PORT_3002_TCP_PORT: 3002
    NITRO_SERVER_PORT_3001_TCP_PORT: 3001
    OFFLINE_PORT_8080_TCP: tcp://172.30.68.82:8080
    WEBAPP_MAINTENANCE_PORT_8080_TCP_PROTO: tcp
    KAFKA_REST_PROXY_PORT_8082_TCP_ADDR: 172.30.248.179
    XERO_PORT: tcp://172.30.89.159:8180
    SIXPACK_WEB_CT_SERVICE_PORT: 8080
    PHPMYADMIN_PORT_80_TCP_PROTO: tcp
    CRATE_EXPORT_PORT_8081_TCP_PORT: 8081
    CRATE_EXPORT_PORT_8080_TCP: tcp://172.30.62.157:8080
    CONNECT_UI_PORT_8090_TCP_ADDR: 172.30.38.156
    KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_CLIENTSTLS: 9093
    KAFKA_SCHEMA_REGISTRY_SERVICE_PORT: 8081
    STRIMZI_VERSION: 0.6.0
    SOFFICE_CONVERT_TO_HTML_PORT_3001_TCP_PROTO: tcp
    FILEBROWSER_PORT_8090_TCP_PROTO: tcp
    HTTP_REDIRECT_DRIVENOW_COM_AU_SERVICE_PORT: 8080
    PIWIK_SERVICE_PORT: 80
    SIXPACK_WEB_CT_PORT: tcp://172.30.99.193:8080
    CRATE_DISCOVERY_SERVICE_PORT_POSTGRES: 5432
    REDIS_SENTINEL_PORT: tcp://172.30.46.154:26379
    DIVOLTE_PORT_8290_TCP_PORT: 8290
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_SERVICE_PORT: 1
    SIXPACK_API_PORT_5000_TCP: tcp://172.30.250.208:5000
    PIWIK_PORT: tcp://172.30.207.150:80
    FILEBROWSER_SERVICE_PORT: 8080
    WEBAPP_PORT_8080_TCP_PORT: 8080
    BLOG_HTTP_PORT_80_TCP: tcp://172.30.42.85:80
    KUBERNETES_PORT_53_TCP_PROTO: tcp
    HTTP_REDIRECT_DRIVENOW_CO_NZ_SERVICE_HOST: 172.30.125.15
    KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://172.30.225.203:9091
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_PORT_1_TCP: tcp://172.30.195.177:1
    CF_TIMEOUT_SERVICE_PORT_8080_TCP: 8080
    XTRADB_SERVICE_PORT: 3306
    KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP: tcp://172.30.225.203:9404
    PHPMYADMIN_SERVICE_HOST: 172.30.59.23
    SFTPD_SERVICE_PORT: 9000
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_PORT_1_TCP_ADDR: 172.30.195.177
    KAFKA_CONNECT_PORT_8083_TCP_PORT: 8083
    SFTPD_PORT_9000_TCP_PORT: 9000
    MYSQL_SERVICE_HOST: 172.30.243.204
    XTRADB_PORT_3306_TCP: tcp://172.30.191.101:3306
    XTRADB_PORT: tcp://172.30.191.101:3306
    KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
    REDIRECT_CAMPERVANHIRENOW_PORT_8080_TCP_PROTO: tcp
    WEBDATA_HTTP_SERVICE_PORT_HTTP: 8080
    KEYCLOAK_PORT_9990_TCP_PORT: 9990
    GLUSTERFS_CLUSTER_PORT_1_TCP_ADDR: 172.30.243.111
    OFFLINE_PORT_8080_TCP_ADDR: 172.30.68.82
    KUBERNETES_PORT_53_TCP_ADDR: 172.30.0.1
    KIBANA_PORT_8090_TCP_ADDR: 172.30.95.178
    HOSTNAME: strimzi-cluster-operator-390472989-9kl9v
    REDIS_SENTINEL_PORT_26379_TCP_PROTO: tcp
    ELASTICSEARCH_SERVICE_PORT_HTTP: 8080
    CRATE_PORT_4300_TCP_PORT: 4300
    KAFKA_REST_PROXY_SERVICE_PORT_8082_TCP: 8082
    ELASTICSEARCH_DISCOVERY_PORT_9300_TCP_PORT: 9300
    REDIS_SENTINEL_PORT_26379_TCP_ADDR: 172.30.46.154
    HOME: /
    MALLOC_ARENA_MAX: 2
    STRIMZI_DEFAULT_ZOOKEEPER_IMAGE: strimzi/zookeeper:0.6.0
    SMTP_RELAY_PORT_25_TCP_PORT: 25
    OSM_CACHE_SERVICE_PORT_HTTP: 80
    KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
    TOPICS_UI_PORT_8090_TCP_PROTO: tcp
    XTRADB_PORT_3306_TCP_PORT: 3306
    KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP_ADDR: 172.30.74.154
    OSM_CACHE_SERVICE_PORT: 80
    MYSQL_PORT_3306_TCP_ADDR: 172.30.243.204
    ELASTICSEARCH_PORT_9200_TCP_PROTO: tcp
    BLOG_2018_HTTP_PORT_80_TCP: tcp://172.30.105.159:80
    DIVOLTE_PORT_8290_TCP_ADDR: 172.30.42.87
    KAFKA_REST_PROXY_PORT: tcp://172.30.248.179:8082
    REDIS_SENTINEL_SERVICE_PORT: 26379
    SIXPACK_WEB_CT_PORT_8080_TCP_PROTO: tcp
    SIXPACK_API_SERVICE_PORT_SIXPACK_API: 5000
    CRATE_DISCOVERY_PORT_5432_TCP: tcp://172.30.184.166:5432
    PIWIK_PORT_80_TCP_PORT: 80
    KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 172.30.225.203
    EMAILQ_PORT_8080_TCP_PROTO: tcp
    THUMBOR_PORT: tcp://172.30.183.243:8000
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_PORT_1_TCP_PROTO: tcp
    SOFFICE_CONVERT_TO_HTML_PORT_3001_TCP_PORT: 3001
    TILESERVER_GL_PORT_80_TCP_PORT: 80
    GLUSTERFS_DYNAMIC_PLUGINS_SERVICE_PORT: 1
    FILEBROWSER_PORT_8080_TCP_PORT: 8080
    SIXPACK_WEB_CT_SERVICE_PORT_HTTP: 8080
    TILESERVER_GL_SERVICE_PORT: 80
    DIVOLTE_SERVICE_PORT: 8290
    KIBANA_PORT_8090_TCP_PORT: 8090
    STRIMZI_NAMESPACE: drivenow-production
    SFTPD_SERVICE_PORT_SFTP: 9000
    KEYCLOAK_SERVICE_HOST: 172.30.88.83
    OSM_CACHE_PORT_80_TCP_ADDR: 172.30.125.26
    NITRO_SERVER_PORT_8080_TCP_ADDR: 172.30.239.225
    TOPICS_UI_SERVICE_PORT: 8090
    PHPMYADMIN_PORT_8080_TCP_PROTO: tcp
    NITRO_SERVER_SERVICE_PORT: 8080
    KUBERNETES_SERVICE_PORT_HTTPS: 443
    SHLVL: 1
    WEBDATA_HTTP_SERVICE_PORT: 8080
    KIBANA_SERVICE_PORT: 8090
    KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
    FILEBROWSER_PORT_8090_TCP: tcp://172.30.73.131:8090
    SFTPD_PORT_9000_TCP_ADDR: 172.30.222.113
    FILEBROWSER_SERVICE_PORT_8080_TCP: 8080
    BLOG_2018_HTTP_SERVICE_PORT_HTTP: 80
    JAVA_HOME: /usr/lib/jvm/java
    HTTP_REDIRECT_DRIVENOW_COM_AU_PORT_8080_TCP: tcp://172.30.182.239:8080
    BLOG_2018_HTTP_PORT_80_TCP_ADDR: 172.30.105.159
    GLUSTERFS_DYNAMIC_PLUGINS_PORT: tcp://172.30.126.79:1
    NITRO_SERVER_PORT_8080_TCP: tcp://172.30.239.225:8080
    TOPICS_UI_PORT_8090_TCP_PORT: 8090
    ELASTICSEARCH_PORT_9200_TCP_ADDR: 172.30.56.77
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_SERVICE_PORT: 1
    CRATE_PORT_4300_TCP_PROTO: tcp
    SIXPACK_API_SERVICE_PORT: 5000
    CF_TIMEOUT_PORT_8080_TCP_ADDR: 172.30.240.149
    REDIRECT_CAMPERVANHIRENOW_PORT: tcp://172.30.154.75:8080
    REDIRECT_CAMPERVANHIRENOW_SERVICE_HOST: 172.30.154.75
    KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
    ELASTICSEARCH_PORT_8080_TCP_PROTO: tcp
    STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: strimzi/kafka-init:0.6.0
    KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP_ADDR: 172.30.8.39
    SMTP_RELAY_SERVICE_HOST: 172.30.253.189
    CONNECT_UI_PORT_8090_TCP: tcp://172.30.38.156:8090
    SCHEMA_REGISTRY_UI_PORT_8090_TCP_PORT: 8090
    BLOG_2018_HTTP_PORT: tcp://172.30.105.159:80
    XERO_PORT_8180_TCP: tcp://172.30.89.159:8180
    CRATE_PORT_8080_TCP_ADDR: 172.30.241.20
    HTTP_REDIRECT_DRIVENOW_COM_AU_SERVICE_HOST: 172.30.182.239
    ELASTICSEARCH_DISCOVERY_SERVICE_PORT_TRANSPORT: 9300
    SMTP_RELAY_PORT_25_TCP: tcp://172.30.253.189:25
    SCHEMA_REGISTRY_UI_PORT: tcp://172.30.228.182:8090
    MYSQL_PORT_3306_TCP_PORT: 3306
    NITRO_SERVER_PORT_8080_TCP_PORT: 8080
    CONNECT_UI_SERVICE_PORT_8090_TCP: 8090
    PHPMYADMIN_PORT_8080_TCP_ADDR: 172.30.59.23
    XTRADB_PORT_3306_TCP_ADDR: 172.30.191.101
    SMTP_RELAY_PORT_25_TCP_PROTO: tcp
    SIXPACK_WEB_SERVICE_HOST: 172.30.100.216
    SOFFICE_CONVERT_TO_HTML_SERVICE_PORT: 3001
    ELASTICSEARCH_PORT: tcp://172.30.56.77:8080
    KUBERNETES_PORT_443_TCP: tcp://172.30.0.1:443
    ELASTICSEARCH_DISCOVERY_PORT: tcp://172.30.182.210:9300
    SIXPACK_REDIS_PORT: tcp://172.30.107.254:6379
    ELASTICSEARCH_PORT_9300_TCP: tcp://172.30.56.77:9300
    KEYCLOAK_PORT_8080_TCP_PROTO: tcp
    SIXPACK_WEB_PORT_8080_TCP: tcp://172.30.100.216:8080
    SFTPD_PORT: tcp://172.30.222.113:9000
    BLOG_HTTP_PORT_80_TCP_PROTO: tcp
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP: tcp://172.30.8.39:1
    SOFFICE_CONVERT_TO_HTML_PORT_3001_TCP_ADDR: 172.30.40.151
    CRATE_DISCOVERY_PORT_4200_TCP_ADDR: 172.30.184.166
    KIBANA_SERVICE_PORT_8090_TCP: 8090
    SIXPACK_REDIS_PORT_6379_TCP_PORT: 6379
    KEYCLOAK_SERVICE_PORT_HTTP_ADMIN: 9990
    CF_TIMEOUT_PORT_8080_TCP_PROTO: tcp
    ELASTICSEARCH_PORT_9200_TCP: tcp://172.30.56.77:9200
    FILEBROWSER_PORT_8090_TCP_PORT: 8090
    BLOG_2018_HTTP_PORT_80_TCP_PORT: 80
    PIWIK_SERVICE_HOST: 172.30.207.150
    CRATE_DISCOVERY_PORT_4200_TCP: tcp://172.30.184.166:4200
    KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_METRICS: 9404
    TILESERVER_GL_SERVICE_PORT_HTTP: 80
    SIXPACK_REDIS_SERVICE_HOST: 172.30.107.254
    ELASTICSEARCH_DISCOVERY_SERVICE_PORT_TRANSPORT9200: 9200
    KAFKA_REST_PROXY_SERVICE_PORT: 8082
    MYSQL_PORT: tcp://172.30.243.204:3306
    KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
    SCHEMA_REGISTRY_UI_PORT_8090_TCP_ADDR: 172.30.228.182
    GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP: tcp://172.30.126.79:1
    KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT_CLIENTS: 2181
    GLUSTERFS_DYNAMIC_WEBDATA_PORT_1_TCP_PORT: 1
    KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP: tcp://172.30.3.22:9404
    SMTP_RELAY_SERVICE_PORT_25_TCP: 25
    STRIMZI_OPERATION_TIMEOUT_MS: 300000
    KAFKA_CONNECT_SERVICE_PORT: 8083
    KAFKA_EXPORTER_SERVICE_HOST: 172.30.229.55
    XTRADB_SERVICE_HOST: 172.30.191.101
    PHPMYADMIN_PORT_80_TCP: tcp://172.30.59.23:80
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT_1_TCP_PORT: 1
    WEBAPP_MAINTENANCE_SERVICE_HOST: 172.30.243.52
    HTTP_REDIRECT_DRIVENOW_CO_NZ_PORT_8080_TCP_PROTO: tcp
    KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP_PROTO: tcp
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_PORT_1_TCP_ADDR: 172.30.211.3
    CONNECT_UI_PORT_8090_TCP_PORT: 8090
    TOPICS_UI_PORT_8090_TCP_ADDR: 172.30.122.9
    MYSQL_SERVICE_PORT: 3306
    CRATE_PORT_4200_TCP_ADDR: 172.30.241.20
    DIVOLTE_PORT_8290_TCP_PROTO: tcp
    KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_REPLICATION: 9091
    LLP_CONVERTER_SERVICE_PORT_HTTP: 3002
    PHPMYADMIN_SERVICE_PORT: 8080
    LLP_CONVERTER_PORT: tcp://172.30.237.96:3002
    CRATE_PORT_8080_TCP_PORT: 8080
    STRIMZI_DEFAULT_KAFKA_IMAGE: strimzi/kafka:0.6.0
    KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 172.30.3.22
    KIBANA_SERVICE_HOST: 172.30.95.178
    KAFKA_CONNECT_PORT_8083_TCP_PROTO: tcp
    WEBDATA_HTTP_SERVICE_HOST: 172.30.12.167
    WEBAPP_PORT: tcp://172.30.205.114:8080
    STRIMZI_DEFAULT_TLS_SIDECAR_KAFKA_IMAGE: strimzi/kafka-stunnel:0.6.0
    SIXPACK_API_PORT_5000_TCP_PROTO: tcp
    ELASTICSEARCH_SERVICE_HOST: 172.30.56.77
    FILEBROWSER_PORT_8080_TCP_PROTO: tcp
    HTTP_REDIRECT_DRIVENOW_COM_AU_PORT_8080_TCP_PROTO: tcp
    ELASTICSEARCH_PORT_8080_TCP: tcp://172.30.56.77:8080
    WEBAPP_SERVICE_HOST: 172.30.205.114
    SOFFICE_CONVERT_TO_HTML_SERVICE_PORT_HTTP: 3001
    SIXPACK_WEB_CT_PORT_8080_TCP_PORT: 8080
    FILEBROWSER_PORT: tcp://172.30.73.131:8080
    GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP_PROTO: tcp
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_PORT_1_TCP: tcp://172.30.211.3:1
    KAFKA_EXPORTER_SERVICE_PORT_9308_TCP: 9308
    XTRADB_SERVICE_PORT_3306_TCP: 3306
    CRATE_EXPORT_PORT_8081_TCP_PROTO: tcp
    CRATE_DISCOVERY_SERVICE_PORT: 4200
    STRIMZI_LOG_LEVEL: DEBUG
    FILEBROWSER_SERVICE_PORT_8090_KEYCLOAK: 8090
    REDIRECT_CAMPERVANHIRENOW_PORT_8080_TCP: tcp://172.30.154.75:8080
    HTTP_REDIRECT_DRIVENOW_CO_NZ_SERVICE_PORT_HTTP: 8080
    KEYCLOAK_PORT_8080_TCP_ADDR: 172.30.88.83
    ELASTICSEARCH_PORT_9200_TCP_PORT: 9200
    REDIRECT_CAMPERVANHIRENOW_PORT_8080_TCP_PORT: 8080
    CRATE_DISCOVERY_PORT_5432_TCP_PROTO: tcp
    KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 172.30.225.203
    KAFKA_EXPORTER_PORT_9308_TCP_PROTO: tcp
    XERO_SERVICE_PORT: 8180
    SCHEMA_REGISTRY_UI_PORT_8090_TCP_PROTO: tcp
    KUBERNETES_PORT: tcp://172.30.0.1:443
    OFFLINE_PORT: tcp://172.30.68.82:8080
    CRATE_DISCOVERY_PORT_5432_TCP_ADDR: 172.30.184.166
    DIVOLTE_SERVICE_HOST: 172.30.42.87
    GLUSTERFS_CLUSTER_PORT_1_TCP: tcp://172.30.243.111:1
    PIWIK_PORT_80_TCP: tcp://172.30.207.150:80
    REDIRECT_CAMPERVANHIRENOW_PORT_8080_TCP_ADDR: 172.30.154.75
    KAFKA_EXPORTER_PORT_9308_TCP_PORT: 9308
    ELASTICSEARCH_DISCOVERY_PORT_9200_TCP_ADDR: 172.30.182.210
    BLOG_HTTP_PORT: tcp://172.30.42.85:80
    SMTP_RELAY_SERVICE_PORT: 25
    NITRO_SERVER_SERVICE_HOST: 172.30.239.225
    SCHEMA_REGISTRY_UI_SERVICE_HOST: 172.30.228.182
    PHPMYADMIN_PORT_80_TCP_ADDR: 172.30.59.23
    CRATE_DISCOVERY_PORT_4300_TCP_PORT: 4300
    EMAILQ_PORT: tcp://172.30.246.123:8080
    CRATE_SERVICE_PORT_PROXY: 8080
    CF_TIMEOUT_PORT_8080_TCP: tcp://172.30.240.149:8080
    SIXPACK_REDIS_PORT_6379_TCP: tcp://172.30.107.254:6379
    CONNECT_UI_SERVICE_PORT: 8090
    SIXPACK_API_PORT: tcp://172.30.250.208:5000
    HTTP_REDIRECT_DRIVENOW_CO_NZ_PORT: tcp://172.30.125.15:8080
    SIXPACK_REDIS_SERVICE_PORT_REDIS: 6379
    SIXPACK_REDIS_PORT_6379_TCP_ADDR: 172.30.107.254
    CF_TIMEOUT_PORT: tcp://172.30.240.149:8080
    WEBAPP_MAINTENANCE_PORT_8080_TCP_ADDR: 172.30.243.52
    KAFKA_ZOOKEEPER_CLIENT_PORT: tcp://172.30.3.22:9404
    MYSQL_PORT_3306_TCP: tcp://172.30.243.204:3306
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_SERVICE_HOST: 172.30.195.177
    MYSQL_PORT_3306_TCP_PROTO: tcp
    KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
    SMTP_RELAY_PORT_25_TCP_ADDR: 172.30.253.189
    REDIS_SENTINEL_PORT_26379_TCP: tcp://172.30.46.154:26379
    CF_TIMEOUT_SERVICE_HOST: 172.30.240.149
    BLOG_2018_HTTP_SERVICE_PORT: 80
    PIWIK_PORT_80_TCP_ADDR: 172.30.207.150
    ELASTICSEARCH_PORT_8080_TCP_PORT: 8080
    CRATE_DISCOVERY_PORT_4300_TCP_ADDR: 172.30.184.166
    THUMBOR_SERVICE_HOST: 172.30.183.243
    STRIMZI_DEFAULT_KAFKA_CONNECT_S2I_IMAGE: strimzi/kafka-connect-s2i:0.6.0
    SIXPACK_REDIS_SERVICE_PORT: 6379
    KAFKA_REST_PROXY_PORT_8082_TCP_PROTO: tcp
    KAFKA_CONNECT_PORT_8083_TCP: tcp://172.30.235.58:8083
    SMTP_RELAY_PORT: tcp://172.30.253.189:25
    KEYCLOAK_PORT_8080_TCP_PORT: 8080
    NITRO_SERVER_PORT: tcp://172.30.239.225:8080
    REDIRECT_CAMPERVANHIRENOW_SERVICE_PORT: 8080
    WEBDATA_HTTP_PORT_8080_TCP_PORT: 8080
    OSM_CACHE_PORT_80_TCP_PORT: 80
    WEBAPP_MAINTENANCE_PORT_8080_TCP: tcp://172.30.243.52:8080
    EMAILQ_SERVICE_PORT_HTTP: 8080
    SOFFICE_CONVERT_TO_HTML_SERVICE_HOST: 172.30.40.151
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_SERVICE_HOST: 172.30.8.39
    KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP: tcp://172.30.74.154:8081
    CRATE_EXPORT_SERVICE_PORT_PROXY: 8081
    KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT: 9404
    WEBDATA_HTTP_PORT_8080_TCP: tcp://172.30.12.167:8080
    SOFFICE_CONVERT_TO_HTML_PORT: tcp://172.30.40.151:3001
    KAFKA_CONNECT_SERVICE_PORT_8083_TCP: 8083
    KUBERNETES_PORT_53_TCP: tcp://172.30.0.1:53
    KAFKA_EXPORTER_SERVICE_PORT: 9308
    FILEBROWSER_PORT_8090_TCP_ADDR: 172.30.73.131
    CRATE_PORT_4200_TCP: tcp://172.30.241.20:4200
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_PORT: tcp://172.30.195.177:1
    SIXPACK_WEB_PORT_8080_TCP_PROTO: tcp
    CRATE_EXPORT_PORT_8080_TCP_ADDR: 172.30.62.157
    SIXPACK_WEB_CT_PORT_8080_TCP_ADDR: 172.30.99.193
    KIBANA_PORT: tcp://172.30.95.178:8090
    KUBERNETES_PORT_53_UDP: udp://172.30.0.1:53
    NITRO_SERVER_PORT_3001_TCP_PROTO: tcp
    WEBAPP_MAINTENANCE_SERVICE_PORT: 8080
    PHPMYADMIN_PORT_8080_TCP_PORT: 8080
    BLOG_2018_HTTP_PORT_80_TCP_PROTO: tcp
    WEBAPP_MAINTENANCE_PORT_8081_TCP_PROTO: tcp
    GLUSTERFS_CLUSTER_PORT_1_TCP_PROTO: tcp
    ELASTICSEARCH_DISCOVERY_SERVICE_HOST: 172.30.182.210
    XTRADB_PORT_3306_TCP_PROTO: tcp
    OFFLINE_SERVICE_PORT_OFFLINE: 8080
    OFFLINE_SERVICE_PORT: 8080
    PHPMYADMIN_PORT_80_TCP_PORT: 80
    CRATE_EXPORT_SERVICE_PORT: 8081
    KUBERNETES_SERVICE_PORT: 443
    CRATE_PORT_8080_TCP_PROTO: tcp
    KAFKA_EXPORTER_PORT: tcp://172.30.229.55:9308
    PHPMYADMIN_PORT_8080_TCP: tcp://172.30.59.23:8080
    LLP_CONVERTER_SERVICE_HOST: 172.30.237.96
    CONNECT_UI_PORT: tcp://172.30.38.156:8090
    BLOG_HTTP_SERVICE_PORT: 80
    KAFKA_EXPORTER_PORT_9308_TCP_ADDR: 172.30.229.55
    KUBERNETES_PORT_53_UDP_ADDR: 172.30.0.1
    HTTP_REDIRECT_DRIVENOW_COM_AU_PORT_8080_TCP_ADDR: 172.30.182.239
    ELASTICSEARCH_PORT_9300_TCP_ADDR: 172.30.56.77
    SFTPD_PORT_9000_TCP_PROTO: tcp
    BLOG_HTTP_PORT_80_TCP_ADDR: 172.30.42.85
    STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: strimzi/topic-operator:0.6.0
    GLUSTERFS_DYNAMIC_PLUGINS_PORT_1_TCP_PORT: 1
    CRATE_DISCOVERY_SERVICE_HOST: 172.30.184.166
    ELASTICSEARCH_SERVICE_PORT_HTTP_PRIVATE: 9200
    WEBDATA_HTTP_PORT_8080_TCP_ADDR: 172.30.12.167
    WEBAPP_PORT_8080_TCP: tcp://172.30.205.114:8080
    WEBAPP_MAINTENANCE_PORT_8081_TCP_ADDR: 172.30.243.52
    ELASTICSEARCH_PORT_9300_TCP_PROTO: tcp
    NITRO_SERVER_PORT_3001_TCP_ADDR: 172.30.239.225
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_PORT: tcp://172.30.211.3:1
    KAFKA_SCHEMA_REGISTRY_PORT: tcp://172.30.74.154:8081
    KIBANA_PORT_8090_TCP: tcp://172.30.95.178:8090
    PWD: /
    ELASTICSEARCH_DISCOVERY_PORT_9300_TCP: tcp://172.30.182.210:9300
    GLUSTERFS_DYNAMIC_WEBDATA_PORT_1_TCP: tcp://172.30.119.83:1
    CRATE_EXPORT_SERVICE_PORT_HTTP: 8080
    NITRO_SERVER_SERVICE_PORT_MANAGEMNENT: 3001
    WEBAPP_MAINTENANCE_PORT_8081_TCP: tcp://172.30.243.52:8081
    KIBANA_PORT_8090_TCP_PROTO: tcp
    ELASTICSEARCH_SERVICE_PORT: 8080
    XERO_SERVICE_HOST: 172.30.89.159
    HTTP_REDIRECT_DRIVENOW_COM_AU_SERVICE_PORT_HTTP: 8080
    GLUSTERFS_DYNAMIC_WEBDATA_PORT: tcp://172.30.119.83:1
    SCHEMA_REGISTRY_UI_PORT_8090_TCP: tcp://172.30.228.182:8090
    CONNECT_UI_PORT_8090_TCP_PROTO: tcp
    ELASTICSEARCH_PORT_8080_TCP_ADDR: 172.30.56.77
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_PORT_1_TCP_PORT: 1
    KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP_PORT: 9404
    CRATE_DISCOVERY_PORT_5432_TCP_PORT: 5432
    EMAILQ_PORT_8080_TCP: tcp://172.30.246.123:8080
    KAFKA_CONNECT_SERVICE_HOST: 172.30.235.58
    WEBAPP_SERVICE_PORT_WEBAPP: 8080
    CRATE_DISCOVERY_PORT_4300_TCP_PROTO: tcp
    SIXPACK_WEB_SERVICE_PORT: 8080
    CONNECT_UI_SERVICE_HOST: 172.30.38.156
    CRATE_DISCOVERY_PORT_4200_TCP_PROTO: tcp
    CRATE_EXPORT_PORT_8080_TCP_PORT: 8080
    DIVOLTE_PORT_8290_TCP: tcp://172.30.42.87:8290
    THUMBOR_PORT_8000_TCP_PROTO: tcp
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_PORT_1_TCP_PROTO: tcp
    WEBAPP_PORT_8080_TCP_PROTO: tcp
    EMAILQ_PORT_8080_TCP_ADDR: 172.30.246.123
    WEBDATA_HTTP_PORT: tcp://172.30.12.167:8080
    OFFLINE_PORT_8080_TCP_PORT: 8080
    KEYCLOAK_PORT_9990_TCP: tcp://172.30.88.83:9990
    KAFKA_KAFKA_BOOTSTRAP_PORT: tcp://172.30.225.203:9091
    EMAILQ_SERVICE_PORT: 8080
    KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://172.30.225.203:9092
    CRATE_PORT_4200_TCP_PROTO: tcp
    KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://172.30.3.22:2181
    KUBERNETES_PORT_443_TCP_ADDR: 172.30.0.1
    ELASTICSEARCH_DISCOVERY_PORT_9300_TCP_ADDR: 172.30.182.210
    THUMBOR_PORT_8000_TCP: tcp://172.30.183.243:8000
    PHPMYADMIN_SERVICE_PORT_HTTP: 8080
    REDIS_SENTINEL_PORT_26379_TCP_PORT: 26379
    STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: strimzi/user-operator:0.6.0
    WEBAPP_MAINTENANCE_SERVICE_PORT_CLUSTER_API: 8081
    GLUSTERFS_DYNAMIC_SUPPLIER_DATA_PORT: tcp://172.30.8.39:1
    ELASTICSEARCH_SERVICE_PORT_TRANSPORT: 9300
    CRATE_DISCOVERY_PORT: tcp://172.30.184.166:4200
    OFFLINE_PORT_8080_TCP_PROTO: tcp
    WEBAPP_PORT_8081_TCP_PORT: 8081
    KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP_PORT: 9404
    KAFKA_CONNECT_PORT_8083_TCP_ADDR: 172.30.235.58
    CRATE_EXPORT_PORT_8081_TCP_ADDR: 172.30.62.157
    SCHEMA_REGISTRY_UI_SERVICE_PORT: 8090
    CRATE_SERVICE_HOST: 172.30.241.20
    KUBERNETES_SERVICE_PORT_DNS_TCP: 53
    TOPICS_UI_PORT: tcp://172.30.122.9:8090
    STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: strimzi/entity-operator-stunnel:0.6.0
    WEBAPP_SERVICE_PORT: 8080
    LLP_CONVERTER_PORT_3002_TCP_ADDR: 172.30.237.96
    KEYCLOAK_PORT_9990_TCP_ADDR: 172.30.88.83
    KAFKA_CONNECT_PORT: tcp://172.30.235.58:8083
    KAFKA_SCHEMA_REGISTRY_SERVICE_PORT_8081_TCP: 8081
    CRATE_PORT_4200_TCP_PORT: 4200
    KAFKA_ZOOKEEPER_CLIENT_PORT_9404_TCP_ADDR: 172.30.3.22
    ELASTICSEARCH_PORT_9300_TCP_PORT: 9300
    THUMBOR_SERVICE_PORT: 8000
    XERO_PORT_8180_TCP_ADDR: 172.30.89.159
    KAFKA_ZOOKEEPER_CLIENT_SERVICE_HOST: 172.30.3.22
    FILEBROWSER_SERVICE_HOST: 172.30.73.131
    GLUSTERFS_DYNAMIC_WEBDATA_PORT_1_TCP_PROTO: tcp
    KUBERNETES_PORT_53_UDP_PORT: 53
    GLUSTERFS_DYNAMIC_BLOG_WP_CONTENT_5G_SERVICE_HOST: 172.30.211.3
    LLP_CONVERTER_PORT_3002_TCP: tcp://172.30.237.96:3002
    SOFFICE_CONVERT_TO_HTML_PORT_3001_TCP: tcp://172.30.40.151:3001
    WEBAPP_MAINTENANCE_SERVICE_PORT_WEBAPP: 8080
    KAFKA_EXPORTER_PORT_9308_TCP: tcp://172.30.229.55:9308
    WEBAPP_PORT_8081_TCP_ADDR: 172.30.205.114
    HTTP_REDIRECT_DRIVENOW_CO_NZ_PORT_8080_TCP_PORT: 8080
    HTTP_REDIRECT_DRIVENOW_COM_AU_PORT_8080_TCP_PORT: 8080
    KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP_ADDR: 172.30.225.203
    LLP_CONVERTER_PORT_3002_TCP_PROTO: tcp
    THUMBOR_PORT_8000_TCP_ADDR: 172.30.183.243
    LLP_CONVERTER_SERVICE_PORT: 3002
    HTTP_REDIRECT_DRIVENOW_CO_NZ_SERVICE_PORT: 8080
    SIXPACK_WEB_CT_SERVICE_HOST: 172.30.99.193
    OSM_CACHE_PORT_80_TCP_PROTO: tcp
    TILESERVER_GL_PORT_80_TCP_PROTO: tcp
    SIXPACK_API_SERVICE_HOST: 172.30.250.208
    CRATE_EXPORT_PORT: tcp://172.30.62.157:8081
    KEYCLOAK_PORT: tcp://172.30.88.83:8080
    GLUSTERFS_DYNAMIC_WEBDATA_SERVICE_PORT: 1
    TOPICS_UI_SERVICE_PORT_8090_TCP: 8090
    ELASTICSEARCH_DISCOVERY_SERVICE_PORT: 9300
    SFTPD_PORT_9000_TCP: tcp://172.30.222.113:9000
    BLOG_HTTP_PORT_80_TCP_PORT: 80
    ELASTICSEARCH_DISCOVERY_PORT_9200_TCP_PORT: 9200
    WEBAPP_MAINTENANCE_PORT_8080_TCP_PORT: 8080
    KEYCLOAK_PORT_9990_TCP_PROTO: tcp
    CF_TIMEOUT_SERVICE_PORT: 8080
    KAFKA_SCHEMA_REGISTRY_SERVICE_HOST: 172.30.74.154
    KUBERNETES_SERVICE_HOST: 172.30.0.1
    GLUSTERFS_CLUSTER_PORT: tcp://172.30.243.111:1
    CRATE_SERVICE_PORT_HTTP: 4200
    EMAILQ_PORT_8080_TCP_PORT: 8080
    SIXPACK_API_PORT_5000_TCP_ADDR: 172.30.250.208
    GLUSTERFS_CLUSTER_SERVICE_HOST: 172.30.243.111
    KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
    KUBERNETES_SERVICE_PORT_DNS: 53
    SIXPACK_WEB_PORT_8080_TCP_ADDR: 172.30.100.216
    SCHEMA_REGISTRY_UI_SERVICE_PORT_8090_TCP: 8090
    CRATE_DISCOVERY_SERVICE_PORT_CRATE_WEB: 4200
    FILEBROWSER_PORT_8080_TCP: tcp://172.30.73.131:8080
    PHPMYADMIN_SERVICE_PORT_HTTP_PRIVATE: 80
    KAFKA_SCHEMA_REGISTRY_PORT_8081_TCP_PROTO: tcp
    ELASTICSEARCH_DISCOVERY_PORT_9200_TCP: tcp://172.30.182.210:9200
    THUMBOR_SERVICE_PORT_HTTP: 8000
    WEBDATA_HTTP_PORT_8080_TCP_PROTO: tcp
    GLUSTERFS_DYNAMIC_BLOG_2018_WP_CONTENT_SERVICE_PORT: 1
    XERO_PORT_8180_TCP_PROTO: tcp
    KAFKA_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://172.30.225.203:9093
    KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
    KEYCLOAK_PORT_8080_TCP: tcp://172.30.88.83:8080
    SIXPACK_WEB_PORT: tcp://172.30.100.216:8080
    KAFKA_KAFKA_BOOTSTRAP_PORT_9404_TCP_PROTO: tcp
    OSM_CACHE_PORT: tcp://172.30.125.26:80
    SFTPD_SERVICE_HOST: 172.30.222.113
    BLOG_2018_HTTP_SERVICE_HOST: 172.30.105.159
    GLUSTERFS_DYNAMIC_PLUGINS_SERVICE_HOST: 172.30.126.79
    WEBAPP_PORT_8081_TCP_PROTO: tcp
    KUBERNETES_PORT_443_TCP_PORT: 443
    STRIMZI_DEFAULT_TLS_SIDECAR_ZOOKEEPER_IMAGE: strimzi/zookeeper-stunnel:0.6.0
2018-11-01 07:51:09 INFO  ClusterOperator:55 - Creating ClusterOperator for namespace drivenow-production
2018-11-01 07:51:09 INFO  ClusterOperator:82 - Starting ClusterOperator for namespace drivenow-production
2018-11-01 07:51:09 DEBUG WatchConnectionManager:101 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@37f7912a
2018-11-01 07:51:09 DEBUG WatchConnectionManager:151 - WebSocket successfully opened
2018-11-01 07:51:09 INFO  ClusterOperator:89 - Started operator for Kafka kind
2018-11-01 07:51:09 DEBUG WatchConnectionManager:101 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@13e160a2
2018-11-01 07:51:09 DEBUG WatchConnectionManager:151 - WebSocket successfully opened
2018-11-01 07:51:09 INFO  ClusterOperator:94 - Started operator for KafkaConnect kind
2018-11-01 07:51:09 DEBUG WatchConnectionManager:101 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5d9b033b
2018-11-01 07:51:09 DEBUG WatchConnectionManager:151 - WebSocket successfully opened
2018-11-01 07:51:09 INFO  ClusterOperator:104 - Started operator for KafkaConnectS2I kind
2018-11-01 07:51:09 INFO  ClusterOperator:107 - Setting up periodical reconciliation for namespace drivenow-production
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.numHeapArenas: 2
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.numDirectArenas: 2
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.pageSize: 8192
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.maxOrder: 11
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.chunkSize: 16777216
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.tinyCacheSize: 512
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.smallCacheSize: 256
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.normalCacheSize: 64
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.cacheTrimInterval: 8192
2018-11-01 07:51:10 DEBUG PooledByteBufAllocator:76 - -Dio.netty.allocator.useCacheForAllThreads: true
2018-11-01 07:51:10 DEBUG InternalThreadLocalMap:76 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2018-11-01 07:51:10 DEBUG InternalThreadLocalMap:76 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2018-11-01 07:51:10 INFO  AbstractAssemblyOperator:352 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): Kafka kafka in namespace drivenow-production was ADDED
2018-11-01 07:51:10 DEBUG AbstractAssemblyOperator:220 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): Lock lock::drivenow-production::Kafka::kafka acquired
2018-11-01 07:51:10 DEBUG DefaultChannelId:76 - -Dio.netty.processId: 1 (auto-detected)
2018-11-01 07:51:10 DEBUG DefaultChannelId:76 - -Dio.netty.machineId: 0a:58:0a:ff:fe:82:01:c4 (auto-detected)
2018-11-01 07:51:10 INFO  AbstractAssemblyOperator:228 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): Assembly kafka should be created or updated
2018-11-01 07:51:10 DEBUG ByteBufUtil:76 - -Dio.netty.allocator.type: pooled
2018-11-01 07:51:10 DEBUG ByteBufUtil:76 - -Dio.netty.threadLocalDirectBufferSize: 65536
2018-11-01 07:51:10 DEBUG ByteBufUtil:76 - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2018-11-01 07:51:10 INFO  ClusterOperator:160 - ClusterOperator is now ready (health server listening on 8080)
2018-11-01 07:51:10 DEBUG AbstractAssemblyOperator:164 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): The cluster CA kafka-cluster-ca already exists
2018-11-01 07:51:10 INFO  Main:114 - Cluster Operator verticle started in namespace drivenow-production
2018-11-01 07:51:10 DEBUG SecretOperator:87 - Secret drivenow-production/kafka-cluster-ca already exists, patching it
2018-11-01 07:51:10 DEBUG SecretOperator:130 - Secret kafka-cluster-ca in namespace drivenow-production has been patched
2018-11-01 07:51:10 DEBUG KafkaAssemblyOperator:517 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): create/update zookeeper kafka
2018-11-01 07:51:10 DEBUG AbstractModel:188 - Generating certificates
2018-11-01 07:51:10 DEBUG AbstractModel:206 - Cluster communication certificates
2018-11-01 07:51:10 DEBUG AbstractModel:1038 - kafka-zookeeper-0 already exists
2018-11-01 07:51:10 DEBUG AbstractModel:1038 - kafka-zookeeper-1 already exists
2018-11-01 07:51:10 DEBUG AbstractModel:1038 - kafka-zookeeper-2 already exists
2018-11-01 07:51:10 DEBUG AbstractModel:215 - End generating certificates
2018-11-01 07:51:11 DEBUG ServiceOperator:87 - Service drivenow-production/kafka-zookeeper-client already exists, patching it
2018-11-01 07:51:11 DEBUG ServiceOperator:130 - Service kafka-zookeeper-client in namespace drivenow-production has been patched
2018-11-01 07:51:11 DEBUG ServiceOperator:87 - Service drivenow-production/kafka-zookeeper-nodes already exists, patching it
2018-11-01 07:51:11 DEBUG ServiceOperator:130 - Service kafka-zookeeper-nodes in namespace drivenow-production has been patched
2018-11-01 07:51:11 DEBUG ConfigMapOperator:87 - ConfigMap drivenow-production/kafka-zookeeper-config already exists, patching it
2018-11-01 07:51:11 DEBUG ConfigMapOperator:47 - ConfigMap kafka-zookeeper-config in namespace drivenow-production has not been patched because resources are equal
2018-11-01 07:51:11 DEBUG SecretOperator:87 - Secret drivenow-production/kafka-zookeeper-nodes already exists, patching it
2018-11-01 07:51:11 DEBUG SecretOperator:130 - Secret kafka-zookeeper-nodes in namespace drivenow-production has been patched
2018-11-01 07:51:11 DEBUG ZookeeperSetOperator:87 - StatefulSet drivenow-production/kafka-zookeeper already exists, patching it
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/revisionHistoryLimit"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/metadata/annotations"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/imagePullPolicy"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/successThreshold"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/successThreshold"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/resources"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePath"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePolicy"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/imagePullPolicy"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/resources"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePath"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePolicy"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/dnsPolicy"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/restartPolicy"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/schedulerName"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/securityContext"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/terminationGracePeriodSeconds"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/0/configMap/defaultMode"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/1/secret/defaultMode"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/status"}
2018-11-01 07:51:11 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-zookeeper ignoring diff {"op":"remove","path":"/status"}
2018-11-01 07:51:11 DEBUG StatefulSetOperator:242 - Patching StatefulSet drivenow-production/kafka-zookeeper
2018-11-01 07:51:11 DEBUG StatefulSetOperator:245 - Patched StatefulSet drivenow-production/kafka-zookeeper
2018-11-01 07:51:11 DEBUG StatefulSetOperator:77 - Considering rolling update of drivenow-production/kafka-zookeeper
2018-11-01 07:51:12 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-production/kafka-zookeeper: pod kafka-zookeeper-0 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
2018-11-01 07:51:12 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-production/kafka-zookeeper: pod kafka-zookeeper-0 has operator.strimzi.io/statefulset-generation=0; no need to roll
2018-11-01 07:51:12 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-production/kafka-zookeeper: pod kafka-zookeeper-1 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
2018-11-01 07:51:12 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-production/kafka-zookeeper: pod kafka-zookeeper-1 has operator.strimzi.io/statefulset-generation=0; no need to roll
2018-11-01 07:51:12 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-production/kafka-zookeeper: pod kafka-zookeeper-2 has operator.strimzi.io/statefulset-generation=0; ss has operator.strimzi.io/statefulset-generation=0
2018-11-01 07:51:12 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-production/kafka-zookeeper: pod kafka-zookeeper-2 has operator.strimzi.io/statefulset-generation=0; no need to roll
2018-11-01 07:51:12 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-zookeeper-client in namespace drivenow-production to get ready
2018-11-01 07:51:12 DEBUG EndpointOperator:88 - Endpoints kafka-zookeeper-client in namespace drivenow-production is ready
2018-11-01 07:51:12 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-zookeeper-nodes in namespace drivenow-production to get ready
2018-11-01 07:51:12 DEBUG EndpointOperator:88 - Endpoints kafka-zookeeper-nodes in namespace drivenow-production is ready
2018-11-01 07:51:12 DEBUG KafkaAssemblyOperator:420 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): create/update kafka kafka
2018-11-01 07:51:12 WARN  AbstractConfiguration:137 - Configuration option "zookeeper.connection.timeout.ms" is forbidden and will be ignored
2018-11-01 07:51:12 DEBUG AbstractModel:248 - Generating certificates
2018-11-01T07:51:12.210+0000: [GC (Allocation Failure) [PSYoungGen: 65536K->10744K(76288K)] 80185K->28436K(251392K), 0.0949875 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 
2018-11-01 07:51:12 DEBUG AbstractModel:280 - Clients CA already exists
2018-11-01 07:51:12 DEBUG AbstractModel:292 - Internal communication certificates
2018-11-01 07:51:12 DEBUG AbstractModel:1038 - kafka-kafka-0 already exists
2018-11-01 07:51:12 DEBUG AbstractModel:1038 - kafka-kafka-1 already exists
2018-11-01 07:51:12 DEBUG AbstractModel:1038 - kafka-kafka-2 already exists
2018-11-01 07:51:12 DEBUG AbstractModel:302 - End generating certificates
2018-11-01 07:51:12 DEBUG ServiceAccountOperator:87 - ServiceAccount drivenow-production/kafka-kafka already exists, patching it
2018-11-01 07:51:12 DEBUG ClusterRoleBindingOperator:107 - Making GET request Request{method=GET, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/strimzi-kafka-kafka-init, tag=null}
2018-11-01 07:51:12 DEBUG ClusterRoleBindingOperator:110 - Got GET response Response{protocol=http/1.1, code=404, message=Not Found, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/strimzi-kafka-kafka-init}
2018-11-01 07:51:12 DEBUG ClusterRoleBindingOperator:76 - No-op (deletion requested, but does not exist)
2018-11-01 07:51:12 DEBUG ServiceOperator:87 - Service drivenow-production/kafka-kafka-bootstrap already exists, patching it
2018-11-01 07:51:12 DEBUG ServiceOperator:130 - Service kafka-kafka-bootstrap in namespace drivenow-production has been patched
2018-11-01 07:51:12 DEBUG ServiceOperator:87 - Service drivenow-production/kafka-kafka-brokers already exists, patching it
2018-11-01 07:51:12 DEBUG ServiceOperator:130 - Service kafka-kafka-brokers in namespace drivenow-production has been patched
2018-11-01 07:51:12 DEBUG ConfigMapOperator:87 - ConfigMap drivenow-production/kafka-kafka-config already exists, patching it
2018-11-01 07:51:12 DEBUG ConfigMapOperator:47 - ConfigMap kafka-kafka-config in namespace drivenow-production has not been patched because resources are equal
2018-11-01 07:51:12 DEBUG SecretOperator:87 - Secret drivenow-production/kafka-clients-ca already exists, patching it
2018-11-01 07:51:12 DEBUG SecretOperator:130 - Secret kafka-clients-ca in namespace drivenow-production has been patched
2018-11-01 07:51:12 DEBUG SecretOperator:87 - Secret drivenow-production/kafka-clients-ca-cert already exists, patching it
2018-11-01 07:51:12 DEBUG SecretOperator:130 - Secret kafka-clients-ca-cert in namespace drivenow-production has been patched
2018-11-01 07:51:12 DEBUG SecretOperator:87 - Secret drivenow-production/kafka-cluster-ca-cert already exists, patching it
2018-11-01 07:51:12 DEBUG SecretOperator:130 - Secret kafka-cluster-ca-cert in namespace drivenow-production has been patched
2018-11-01 07:51:12 DEBUG SecretOperator:87 - Secret drivenow-production/kafka-kafka-brokers already exists, patching it
2018-11-01 07:51:12 DEBUG SecretOperator:130 - Secret kafka-kafka-brokers in namespace drivenow-production has been patched
2018-11-01 07:51:12 DEBUG KafkaSetOperator:87 - StatefulSet drivenow-production/kafka-kafka already exists, patching it
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/revisionHistoryLimit"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/metadata/annotations"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/imagePullPolicy"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/successThreshold"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/successThreshold"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/resources"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePath"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePolicy"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/imagePullPolicy"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/resources"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePath"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePolicy"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/dnsPolicy"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/restartPolicy"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/schedulerName"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/securityContext"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/serviceAccount"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/terminationGracePeriodSeconds"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/0/secret/defaultMode"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/1/secret/defaultMode"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/2/configMap/defaultMode"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/status"}
2018-11-01 07:51:12 DEBUG ResourceOperatorSupplier$StatefulSetDiff:146 - StatefulSet drivenow-production/kafka-kafka ignoring diff {"op":"remove","path":"/status"}
2018-11-01 07:51:12 DEBUG StatefulSetOperator:242 - Patching StatefulSet drivenow-production/kafka-kafka
2018-11-01 07:51:12 DEBUG StatefulSetOperator:245 - Patched StatefulSet drivenow-production/kafka-kafka
2018-11-01 07:51:12 DEBUG StatefulSetOperator:77 - Considering rolling update of drivenow-production/kafka-kafka
2018-11-01 07:51:12 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-production/kafka-kafka: pod kafka-kafka-0 has operator.strimzi.io/statefulset-generation=1; ss has operator.strimzi.io/statefulset-generation=1
2018-11-01 07:51:12 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-production/kafka-kafka: pod kafka-kafka-0 has operator.strimzi.io/statefulset-generation=1; no need to roll
2018-11-01 07:51:13 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-production/kafka-kafka: pod kafka-kafka-1 has operator.strimzi.io/statefulset-generation=1; ss has operator.strimzi.io/statefulset-generation=1
2018-11-01 07:51:13 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-production/kafka-kafka: pod kafka-kafka-1 has operator.strimzi.io/statefulset-generation=1; no need to roll
2018-11-01 07:51:13 DEBUG StatefulSetOperator:134 - Rolling update of drivenow-production/kafka-kafka: pod kafka-kafka-2 has operator.strimzi.io/statefulset-generation=1; ss has operator.strimzi.io/statefulset-generation=1
2018-11-01 07:51:13 DEBUG StatefulSetOperator:93 - Rolling update of drivenow-production/kafka-kafka: pod kafka-kafka-2 has operator.strimzi.io/statefulset-generation=1; no need to roll
2018-11-01 07:51:13 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-kafka-bootstrap in namespace drivenow-production to get ready
2018-11-01 07:51:13 DEBUG EndpointOperator:88 - Endpoints kafka-kafka-bootstrap in namespace drivenow-production is ready
2018-11-01 07:51:13 DEBUG EndpointOperator:66 - Waiting for Endpoints resource kafka-kafka-brokers in namespace drivenow-production to get ready
2018-11-01 07:51:13 DEBUG EndpointOperator:88 - Endpoints kafka-kafka-brokers in namespace drivenow-production is ready
2018-11-01 07:51:13 DEBUG KafkaAssemblyOperator:600 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): create/update topic operator kafka
2018-11-01 07:51:13 DEBUG ServiceAccountOperator:96 - ServiceAccount drivenow-production/kafka-topic-operator does not exist, noop
2018-11-01 07:51:13 DEBUG RoleBindingOperator:107 - Making GET request Request{method=GET, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings/strimzi-topic-operator-role-binding, tag=null}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:110 - Got GET response Response{protocol=http/1.1, code=404, message=Not Found, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings/strimzi-topic-operator-role-binding}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:76 - No-op (deletion requested, but does not exist)
2018-11-01 07:51:13 DEBUG ConfigMapOperator:96 - ConfigMap drivenow-production/kafka-topic-operator-config does not exist, noop
2018-11-01 07:51:13 DEBUG DeploymentOperator:96 - Deployment drivenow-production/kafka-topic-operator does not exist, noop
2018-11-01 07:51:13 DEBUG SecretOperator:96 - Secret drivenow-production/kafka-topic-operator-certs does not exist, noop
2018-11-01 07:51:13 DEBUG KafkaAssemblyOperator:687 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): create/update entity operator kafka
2018-11-01 07:51:13 DEBUG AbstractModel:164 - Generating certificates
2018-11-01 07:51:13 DEBUG AbstractModel:177 - Entity Operator certificate to generate
2018-11-01 07:51:13 DEBUG OpenSslCertManager:225 - Running command [openssl, req, -new, -batch, -nodes, -keyout, /tmp/tls2046774837895143651key, -out, /tmp/tls6818768701944850667csr, -subj, /O=io.strimzi/CN=kafka-entity-operator]
2018-11-01 07:51:13 DEBUG OpenSslCertManager:236 - Generating a 2048 bit RSA private key
................+++
.........................................................+++
unable to write 'random state'
writing new private key to '/tmp/tls2046774837895143651key'
-----
2018-11-01 07:51:13 DEBUG OpenSslCertManager:237 - result 0
2018-11-01 07:51:13 DEBUG OpenSslCertManager:225 - Running command [openssl, x509, -req, -days, 365, -in, /tmp/tls6818768701944850667csr, -CA, /tmp/ca-crt-5964548147317281868.crt, -CAkey, /tmp/ca-key-4379354712821178753.key, -CAcreateserial, -out, /tmp/tls5620446224378214541cert]
2018-11-01 07:51:13 DEBUG OpenSslCertManager:236 - Signature ok
subject=/O=io.strimzi/CN=kafka-entity-operator
Getting CA Private Key
unable to write 'random state'
2018-11-01 07:51:13 DEBUG OpenSslCertManager:237 - result 0
2018-11-01 07:51:13 DEBUG AbstractModel:205 - End generating certificates
2018-11-01 07:51:13 DEBUG ServiceAccountOperator:87 - ServiceAccount drivenow-production/kafka-entity-operator already exists, patching it
2018-11-01 07:51:13 DEBUG RoleBindingOperator:107 - Making GET request Request{method=GET, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings/strimzi-entity-topic-operator-role-binding, tag=null}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:110 - Got GET response Response{protocol=http/1.1, code=404, message=Not Found, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings/strimzi-entity-topic-operator-role-binding}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:72 - Creating
2018-11-01 07:51:13 DEBUG RoleBindingOperator:107 - Making POST request Request{method=POST, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings, tag=null}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:110 - Got POST response Response{protocol=http/1.1, code=403, message=Forbidden, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings}
2018-11-01 07:51:13 DEBUG AbstractAssemblyOperator:239 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): Lock lock::drivenow-production::Kafka::kafka released
2018-11-01 07:51:13 ERROR AbstractAssemblyOperator:244 - Reconciliation #0(watch) Kafka(drivenow-production/kafka): createOrUpdate failed
io.fabric8.kubernetes.client.KubernetesClientException: Got unexpected POST status code 403: Forbidden
    at io.strimzi.operator.common.operator.resource.WorkaroundRbacOperator.execute(WorkaroundRbacOperator.java:118) ~[cluster-operator-0.6.0.jar:?]
    at io.strimzi.operator.common.operator.resource.WorkaroundRbacOperator.create(WorkaroundRbacOperator.java:145) ~[cluster-operator-0.6.0.jar:?]
    at io.strimzi.operator.common.operator.resource.WorkaroundRbacOperator.lambda$doReconcile$0(WorkaroundRbacOperator.java:73) ~[cluster-operator-0.6.0.jar:?]
    at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:273) ~[cluster-operator-0.6.0.jar:?]
    at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[cluster-operator-0.6.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [cluster-operator-0.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-11-01 07:51:18 DEBUG Recycler:76 - -Dio.netty.recycler.maxCapacityPerThread: 32768
2018-11-01 07:51:18 DEBUG Recycler:76 - -Dio.netty.recycler.maxSharedCapacityFactor: 2
2018-11-01 07:51:18 DEBUG Recycler:76 - -Dio.netty.recycler.linkCapacity: 16
2018-11-01 07:51:18 DEBUG Recycler:76 - -Dio.netty.recycler.ratio: 8
2018-11-01 07:51:18 DEBUG AbstractByteBuf:81 - -Dio.netty.buffer.bytebuf.checkAccessible: true
2018-11-01 07:51:18 DEBUG ResourceLeakDetectorFactory:76 - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5ad0d1d6

[scholzj]

Hmm, this is where it seems to fail:

2018-11-01 07:51:13 DEBUG RoleBindingOperator:107 - Making GET request Request{method=GET, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings/strimzi-entity-topic-operator-role-binding, tag=null}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:110 - Got GET response Response{protocol=http/1.1, code=404, message=Not Found, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings/strimzi-entity-topic-operator-role-binding}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:72 - Creating
2018-11-01 07:51:13 DEBUG RoleBindingOperator:107 - Making POST request Request{method=POST, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings, tag=null}
2018-11-01 07:51:13 DEBUG RoleBindingOperator:110 - Got POST response Response{protocol=http/1.1, code=403, message=Forbidden, url=https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1beta1/namespaces/drivenow-production/rolebindings}

It tries to create the role binding for the EntityOperator but it is not allowed to. It is unfortunately never clear why exactly that might be happening. Can you list your cluster roles and cluster role bindings and double check that they are the same as in the 0.6.0 installation files?

[cameronbraid]

I just re-applied them with

oc -n drivenow-production apply -f examples/install/cluster-operator/04-ClusterRoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml
oc -n drivenow-production apply -f examples/install/cluster-operator/04-ClusterRoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml

and now the operator starts up without an error

Thanks for your help

[scholzj]

Ok, great. I will close this issue then. Feel free to reopen it or open a new one if you have something else. Thanks.