search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.89k stars 1.31k forks source link

Cluster crashes when upgrading AKS kubernetes version #4668

Closed msts1906 closed 3 years ago

msts1906 commented 3 years ago

Edit: Provided full logs

Hi everyone,

I am facing the following problem with AKS and strimzi. Thanks in advance for your help!

Describe the bug When upgrading our AKS kubernetes version from 1.16 to 1.17 the cluster stopped working. The kafka pods and one zookeeper pod are trapped in a crashLoop as is the entity operator.

To Reproduce Steps to reproduce the behavior:

  1. aks upgrade --resource-group XXXX --name XXXX --kubernetes-version 1.17.16

Expected behavior The cluster restarts itself and continuous to work as expected

Environment (please complete the following information):

YAML files and logs

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka
spec:
  kafka:
    version: 2.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.7"
      inter.broker.protocol.version: "2.7"
    storage:
      type: persistent-claim
      size: 4Gi
      deleteClaim: false
      class: azurefile
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 1Gi
      deleteClaim: false
      class: azurefile
  entityOperator:
    topicOperator: { }
    userOperator: { }

Logs from Kafka entity operator: `Preparing certificates for internal communication Adding /etc/tls-sidecar/cluster-ca-certs/ca.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca Certificate was added to keystore Preparing certificates for internal communication is complete

2021-03-29 17:23:59,81886 INFO [vert.x-eventloop-thread-0] Session:149 - Starting 2021-03-29 17:24:00,18256 INFO [vert.x-eventloop-thread-0] AppInfoParser:119 - Kafka version: 2.7.0 2021-03-29 17:24:00,18290 INFO [vert.x-eventloop-thread-0] AppInfoParser:120 - Kafka commitId: 448719dc99a19793 2021-03-29 17:24:00,18318 INFO [vert.x-eventloop-thread-0] AppInfoParser:121 - Kafka startTimeMs: 1617038640179 2021-03-29 17:24:00,27715 INFO [ZkClient-EventThread-19-localhost:2181] ZkEventThread:65 - Starting ZkClient event thread. 2021-03-29 17:24:00,28232 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT 2021-03-29 17:24:00,28284 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:host.name=kafka-entity-operator-5868cd576f-l9cf8 2021-03-29 17:24:00,28306 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.version=11.0.10 2021-03-29 17:24:00,28325 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.vendor=Red Hat, Inc. 2021-03-29 17:24:00,28342 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.10.0.9-0.el7_9.x86_64 2021-03-29 17:24:00,28362 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.class.path=lib/io.strimzi.topic-operator-0.22.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.7.jar:lib/io.fabric8.kubernetes-model-storageclass-5.0.2.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/io.fabric8.openshift-model-operator-5.0.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.60.Final.jar:lib/com.google.api.grpc.proto-google-common-protos-1.17.0.jar:lib/org.apache.kafka.connect-api-2.7.0.jar:lib/io.fabric8.kubernetes-model-settings-5.0.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.google.j2objc.j2objc-annotations-1.3.jar:lib/io.apicurio.apicurio-registry-common-1.3.0.Final.jar:lib/io.fabric8.kubernetes-model-coordination-5.0.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.netty.netty-codec-4.1.60.Final.jar:lib/org.codehaus.mojo.animal-sniffer-annotations-1.18.jar:lib/io.netty.netty-codec-http2-4.1.60.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.0.2.jar:lib/io.grpc.grpc-context-1.31.1.jar:lib/io.fabric8.openshift-client-5.0.2.jar:lib/io.fabric8.openshift-model-console-5.0.2.jar:lib/io.grpc.grpc-protobuf-lite-1.31.1.jar:lib/io.strimzi.operator-common-0.22.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.0.Final.jar:lib/io.perfmark.perfmark-api-0.19.0.jar:lib/io.fabric8.kubernetes-model-rbac-5.0.2.jar:lib/io.fabric8.kubernetes-model-policy-5.0.2.jar:lib/io.netty.netty-codec-http-4.1.60.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.5.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.openshift-model-operatorhub-5.0.2.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.grpc.grpc-api-1.30.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/io.fabric8.kubernetes-model-discovery-5.0.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.5.1.jar:lib/io.fabric8.openshift-model-5.0.2.jar:lib/io.grpc.grpc-netty-shaded-1.31.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar:lib/io.fabric8.kubernetes-model-node-5.0.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.0.2.jar:lib/io.grpc.grpc-protobuf-1.31.1.jar:lib/com.google.android.annotations-4.1.1.4.jar:lib/io.fabric8.kubernetes-client-5.0.2.jar:lib/org.rocksdb.rocksdbjni-5.18.4.jar:lib/io.netty.netty-common-4.1.60.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.fabric8.kubernetes-model-apps-5.0.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/com.github.luben.zstd-jni-1.4.5-6.jar:lib/io.netty.netty-codec-socks-4.1.60.Final.jar:lib/io.netty.netty-transport-4.1.60.Final.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.0.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.60.Final.jar:lib/io.netty.netty-handler-proxy-4.1.60.Final.jar:lib/io.fabric8.openshift-model-monitoring-5.0.2.jar:lib/com.101tec.zkclient-0.11.jar:lib/com.google.guava.listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.checkerframework.checker-compat-qual-2.5.5.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.google.guava.failureaccess-1.0.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.5.jar:lib/io.fabric8.kubernetes-model-batch-5.0.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.apache.kafka.kafka-clients-2.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.60.Final.jar:lib/io.netty.netty-buffer-4.1.60.Final.jar:lib/com.google.protobuf.protobuf-java-3.13.0.jar:lib/io.netty.netty-handler-4.1.60.Final.jar:lib/org.apache.kafka.kafka-streams-2.6.0.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.google.code.findbugs.jsr305-3.0.2.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.0.Final.jar:lib/io.fabric8.kubernetes-model-scheduling-5.0.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.5.jar:lib/io.fabric8.kubernetes-model-extensions-5.0.2.jar:lib/com.google.errorprone.error_prone_annotations-2.3.4.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/io.fabric8.kubernetes-model-events-5.0.2.jar:lib/io.grpc.grpc-core-1.31.1.jar:lib/com.google.guava.guava-28.2-android.jar:lib/io.fabric8.kubernetes-model-core-5.0.2.jar:lib/io.strimzi.crd-annotations-0.22.0.jar:lib/io.fabric8.kubernetes-model-networking-5.0.2.jar:lib/io.strimzi.certificate-manager-0.22.0.jar:lib/io.grpc.grpc-stub-1.31.1.jar:lib/io.fabric8.kubernetes-model-certificates-5.0.2.jar:lib/io.fabric8.kubernetes-model-common-5.0.2.jar:lib/io.netty.netty-resolver-4.1.60.Final.jar:lib/org.apache.kafka.connect-json-2.7.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.api-0.22.0.jar:lib/io.fabric8.kubernetes-model-metrics-5.0.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.netty.netty-codec-dns-4.1.60.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar 2021-03-29 17:24:00,28422 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2021-03-29 17:24:00,28445 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.io.tmpdir=/tmp 2021-03-29 17:24:00,28465 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:java.compiler= 2021-03-29 17:24:00,28486 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:os.name=Linux 2021-03-29 17:24:00,28505 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:os.arch=amd64 2021-03-29 17:24:00,28523 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:os.version=4.15.0-1109-azure 2021-03-29 17:24:00,28543 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:user.name=strimzi 2021-03-29 17:24:00,28563 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:user.home=/home/strimzi 2021-03-29 17:24:00,28602 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:user.dir=/opt/strimzi 2021-03-29 17:24:00,28631 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:os.memory.free=186MB 2021-03-29 17:24:00,28648 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:os.memory.max=3234MB 2021-03-29 17:24:00,28665 INFO [vert.x-worker-thread-0] ZooKeeper:109 - Client environment:os.memory.total=203MB 2021-03-29 17:24:00,31758 INFO [vert.x-worker-thread-0] ZooKeeper:868 - Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.I0Itec.zkclient.ZkClient@34039d20 2021-03-29 17:24:00,32190 INFO [vert.x-worker-thread-0] X509Util:79 - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2021-03-29 17:24:00,32557 INFO [vert.x-worker-thread-0] ClientCnxnSocket:237 - jute.maxbuffer value is 4194304 Bytes 2021-03-29 17:24:00,33257 INFO [vert.x-worker-thread-0] ClientCnxn:1653 - zookeeper.request.timeout value is 0. feature enabled= 2021-03-29 17:24:00,33348 INFO [vert.x-worker-thread-0] ZkClient:936 - Waiting for keeper state SyncConnected 2021-03-29 17:24:00,33903 INFO [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1112 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-03-29 17:24:00,34043 INFO [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:959 - Socket connection established, initiating session, client: /127.0.0.1:59442, server: localhost/127.0.0.1:2181 2021-03-29 17:24:00,39252 INFO [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1394 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1000005e0e700af, negotiated timeout = 20000 2021-03-29 17:24:00,39440 INFO [vert.x-worker-thread-0-EventThread] ZkClient:713 - zookeeper state changed (SyncConnected) 2021-03-29 17:24:00,52598 INFO [vert.x-eventloop-thread-0] AppInfoParser:119 - Kafka version: 2.7.0 2021-03-29 17:24:00,52631 INFO [vert.x-eventloop-thread-0] AppInfoParser:120 - Kafka commitId: 448719dc99a19793 2021-03-29 17:24:00,52659 INFO [vert.x-eventloop-thread-0] AppInfoParser:121 - Kafka startTimeMs: 1617038640525 2021-03-29 17:24:00,52740 INFO [vert.x-eventloop-thread-0] KafkaStreamsTopicStoreService:51 - Starting ... Mar 29, 2021 5:24:02 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2146 ms, time limit is 2000 ms Mar 29, 2021 5:24:03 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3147 ms, time limit is 2000 ms Mar 29, 2021 5:24:04 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4146 ms, time limit is 2000 ms Mar 29, 2021 5:24:05 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5147 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:06 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 6148 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:07 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 7148 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:08 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 8149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:09 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 9149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:10 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 10148 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:11 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 11148 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:12 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 12149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:13 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 13149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:14 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 14149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:15 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 15149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:16 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 16149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:17 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 17149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:18 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 18149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:19 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 19149 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:20 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 20150 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:21 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 21151 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:22 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 22151 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:23 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 23152 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:24 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 24152 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:25 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 25152 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:26 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 26153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:27 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 27153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:28 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 28153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:29 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 29153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:30 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 30152 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:31 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 31152 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:32 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 32152 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:33 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 33153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:34 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 34153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:35 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 35153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

Mar 29, 2021 5:24:36 PM io.vertx.core.impl.BlockedThreadChecker WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main]=Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 36153 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) at app//io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) at app//io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) at app//io.strimzi.operator.topic.Session.lambda$start$7(Session.java:204) at app//io.strimzi.operator.topic.Session$$Lambda$241/0x000000084028c840.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) at app//io.vertx.core.impl.ContextImpl$$Lambda$251/0x00000008402afc40.handle(Unknown Source) at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) at app//io.vertx.core.impl.EventLoopContext$$Lambda$193/0x0000000840231040.run(Unknown Source) at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.10/java.lang.Thread.run(Thread.java:834) `

Logs from one of the kafka pods: group.max.size = 2147483647 group.min.session.timeout.ms = 6000 host.name = inter.broker.listener.name = REPLICATION-9091 inter.broker.protocol.version = 2.7 kafka.metrics.polling.interval.secs = 10 kafka.metrics.reporters = [] leader.imbalance.check.interval.seconds = 300 leader.imbalance.per.broker.percentage = 10 listener.security.protocol.map = REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT listeners = REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092 log.cleaner.backoff.ms = 15000 log.cleaner.dedupe.buffer.size = 134217728 log.cleaner.delete.retention.ms = 86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9 log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807 log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 1 log.cleanup.policy = [delete] log.dir = /tmp/kafka-logs log.dirs = /var/lib/kafka/data/kafka-log0 log.flush.interval.messages = 9223372036854775807 log.flush.interval.ms = null log.flush.offset.checkpoint.interval.ms = 60000 log.flush.scheduler.interval.ms = 9223372036854775807 log.flush.start.offset.checkpoint.interval.ms = 60000 log.index.interval.bytes = 4096 log.index.size.max.bytes = 10485760 log.message.downconversion.enable = true log.message.format.version = 2.7 log.message.timestamp.difference.max.ms = 9223372036854775807 log.message.timestamp.type = CreateTime log.preallocate = false log.retention.bytes = -1 log.retention.check.interval.ms = 300000 log.retention.hours = 168 log.retention.minutes = null log.retention.ms = null log.roll.hours = 168 log.roll.jitter.hours = 0 log.roll.jitter.ms = null log.roll.ms = null log.segment.bytes = 1073741824 log.segment.delete.delay.ms = 60000 max.connection.creation.rate = 2147483647 max.connections = 2147483647 max.connections.per.ip = 2147483647 max.connections.per.ip.overrides = max.incremental.fetch.session.cache.slots = 1000 message.max.bytes = 1048588 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 min.insync.replicas = 1 num.io.threads = 8 num.network.threads = 3 num.partitions = 1 num.recovery.threads.per.data.dir = 1 num.replica.alter.log.dirs.threads = null num.replica.fetchers = 1 offset.metadata.max.bytes = 4096 offsets.commit.required.acks = -1 offsets.commit.timeout.ms = 5000 offsets.load.buffer.size = 5242880 offsets.retention.check.interval.ms = 600000 offsets.retention.minutes = 10080 offsets.topic.compression.codec = 0 offsets.topic.num.partitions = 50 offsets.topic.replication.factor = 3 offsets.topic.segment.bytes = 104857600 password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding password.encoder.iterations = 4096 password.encoder.key.length = 128 password.encoder.keyfactory.algorithm = null password.encoder.old.secret = null password.encoder.secret = null port = 9092 principal.builder.class = null producer.purgatory.purge.interval.requests = 1000 queued.max.request.bytes = -1 queued.max.requests = 500 quota.consumer.default = 9223372036854775807 quota.producer.default = 9223372036854775807 quota.window.num = 11 quota.window.size.seconds = 1 replica.fetch.backoff.ms = 1000 replica.fetch.max.bytes = 1048576 replica.fetch.min.bytes = 1 replica.fetch.response.max.bytes = 10485760 replica.fetch.wait.max.ms = 500 replica.high.watermark.checkpoint.interval.ms = 5000 replica.lag.time.max.ms = 30000 replica.selector.class = null replica.socket.receive.buffer.bytes = 65536 replica.socket.timeout.ms = 30000 replication.quota.window.num = 11 replication.quota.window.size.seconds = 1 request.timeout.ms = 30000 reserved.broker.max.id = 1000 sasl.client.callback.handler.class = null sasl.enabled.mechanisms = [] sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.principal.to.local.rules = [DEFAULT] sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism.inter.broker.protocol = GSSAPI sasl.server.callback.handler.class = null security.inter.broker.protocol = PLAINTEXT security.providers = null socket.connection.setup.timeout.max.ms = 127000 socket.connection.setup.timeout.ms = 10000 socket.receive.buffer.bytes = 102400 socket.request.max.bytes = 104857600 socket.send.buffer.bytes = 102400 ssl.cipher.suites = [] ssl.client.auth = none ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = HTTPS ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.principal.mapping.rules = DEFAULT ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = SHA1PRNG ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.abort.timed.out.transaction.cleanup.interval.ms = 10000 transaction.max.timeout.ms = 900000 transaction.remove.expired.transaction.cleanup.interval.ms = 3600000 transaction.state.log.load.buffer.size = 5242880 transaction.state.log.min.isr = 1 transaction.state.log.num.partitions = 50 transaction.state.log.replication.factor = 3 transaction.state.log.segment.bytes = 104857600 transactional.id.expiration.ms = 604800000 unclean.leader.election.enable = false zookeeper.clientCnxnSocket = org.apache.zookeeper.ClientCnxnSocketNetty zookeeper.connect = kafka-zookeeper-client:2181 zookeeper.connection.timeout.ms = null zookeeper.max.in.flight.requests = 10 zookeeper.session.timeout.ms = 18000 zookeeper.set.acl = false zookeeper.ssl.cipher.suites = null zookeeper.ssl.client.enable = true zookeeper.ssl.crl.enable = false zookeeper.ssl.enabled.protocols = null zookeeper.ssl.endpoint.identification.algorithm = HTTPS zookeeper.ssl.keystore.location = /tmp/kafka/cluster.keystore.p12 zookeeper.ssl.keystore.password = [hidden] zookeeper.ssl.keystore.type = PKCS12 zookeeper.ssl.ocsp.enable = false zookeeper.ssl.protocol = TLSv1.2 zookeeper.ssl.truststore.location = /tmp/kafka/cluster.truststore.p12 zookeeper.ssl.truststore.password = [hidden] zookeeper.ssl.truststore.type = PKCS12 zookeeper.sync.time.ms = 2000 (kafka.server.KafkaConfig) [main] 2021-03-29 17:25:14,655 INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-Fetch] 2021-03-29 17:25:14,658 INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-Produce] 2021-03-29 17:25:14,660 INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-Request] 2021-03-29 17:25:14,662 INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-ControllerMutation] 2021-03-29 17:25:14,788 INFO Loading logs from log dirs ArrayBuffer(/var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [main] 2021-03-29 17:25:14,795 INFO Attempting recovery for all logs in /var/lib/kafka/data/kafka-log0 since no clean shutdown file was found (kafka.log.LogManager) [main] 2021-03-29 17:25:15,037 INFO [Log partition=assignment-events-0, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:15,039 INFO [Log partition=assignment-events-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:15,487 INFO [Log partition=assignment-events-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:15,577 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/assignment-events-0, topic=assignment-events, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 690ms (1/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:15,659 INFO [Log partition=assignment-events-1, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:15,660 INFO [Log partition=assignment-events-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:16,039 INFO [Log partition=assignment-events-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:16,101 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/assignment-events-1, topic=assignment-events, partition=1, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 523ms (2/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:16,202 INFO [Log partition=assignment-events-2, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:16,203 INFO [Log partition=assignment-events-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:16,479 INFO [Log partition=assignment-events-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:16,553 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/assignment-events-2, topic=assignment-events, partition=2, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 451ms (3/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:16,659 INFO [Log partition=business-logging-0, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:16,659 INFO [Log partition=business-logging-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:17,015 INFO [Log partition=business-logging-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:17,078 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/business-logging-0, topic=business-logging, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 525ms (4/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:17,336 INFO [Log partition=business-logging-1, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:17,336 INFO [Log partition=business-logging-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:17,576 INFO [ProducerStateManager partition=business-logging-1] Writing producer snapshot at offset 1 (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:18,052 INFO [Log partition=business-logging-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 1 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:18,121 INFO [ProducerStateManager partition=business-logging-1] Loading producer state from snapshot file '/var/lib/kafka/data/kafka-log0/business-logging-1/00000000000000000001.snapshot' (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:18,148 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/business-logging-1, topic=business-logging, partition=1, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=1) with 1 segments in 1070ms (5/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:18,232 INFO [Log partition=business-logging-2, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:18,232 INFO [Log partition=business-logging-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:18,710 INFO [Log partition=business-logging-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:18,779 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/business-logging-2, topic=business-logging, partition=2, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 630ms (6/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:18,871 INFO [Log partition=geo-fence-events-0, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:18,872 INFO [Log partition=geo-fence-events-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:19,153 INFO [Log partition=geo-fence-events-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:19,224 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/geo-fence-events-0, topic=geo-fence-events, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 445ms (7/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:19,313 INFO [Log partition=geo-fence-events-1, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:19,313 INFO [Log partition=geo-fence-events-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:19,760 INFO [Log partition=geo-fence-events-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:19,822 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/geo-fence-events-1, topic=geo-fence-events, partition=1, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 598ms (8/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:19,927 INFO [Log partition=geo-fence-events-2, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:19,927 INFO [Log partition=geo-fence-events-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:20,467 INFO [Log partition=geo-fence-events-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:20,527 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/geo-fence-events-2, topic=geo-fence-events, partition=2, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 705ms (9/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:20,605 INFO [Log partition=location-provider-events-0, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:20,605 INFO [Log partition=location-provider-events-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:20,944 INFO [Log partition=location-provider-events-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:21,011 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/location-provider-events-0, topic=location-provider-events, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 484ms (10/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:21,091 INFO [Log partition=location-provider-events-1, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:21,091 INFO [Log partition=location-provider-events-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:21,577 INFO [Log partition=location-provider-events-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:21,646 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/location-provider-events-1, topic=location-provider-events, partition=1, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 634ms (11/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:21,729 INFO [Log partition=location-provider-events-2, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:21,729 INFO [Log partition=location-provider-events-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:22,758 INFO [Log partition=location-provider-events-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:22,832 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/location-provider-events-2, topic=location-provider-events, partition=2, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 1186ms (12/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:22,936 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:22,936 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:23,647 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:23,718 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-0, topic=__consumer_offsets, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 885ms (13/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:23,823 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:23,823 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:24,314 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:24,378 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-1, topic=__consumer_offsets, partition=1, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 659ms (14/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:24,476 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:24,476 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:24,869 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:24,945 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-10, topic=__consumer_offsets, partition=10, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 566ms (15/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:25,031 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:25,032 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:25,685 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:25,748 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-11, topic=__consumer_offsets, partition=11, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 802ms (16/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:25,838 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:25,838 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:26,551 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:26,609 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-12, topic=__consumer_offsets, partition=12, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 862ms (17/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:26,688 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:26,689 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:27,132 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:27,200 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-13, topic=__consumer_offsets, partition=13, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 591ms (18/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:27,289 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:27,289 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:28,448 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:28,532 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-14, topic=__consumer_offsets, partition=14, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 1332ms (19/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:28,675 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:28,675 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:28,983 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:29,045 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-15, topic=__consumer_offsets, partition=15, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 512ms (20/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:29,148 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:29,149 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:29,624 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:29,693 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-16, topic=__consumer_offsets, partition=16, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 647ms (21/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:29,908 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:29,908 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:30,245 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:30,308 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-17, topic=__consumer_offsets, partition=17, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 615ms (22/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:30,393 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:30,394 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:30,766 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:30,837 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-18, topic=__consumer_offsets, partition=18, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 528ms (23/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:30,934 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:30,935 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:31,209 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:31,280 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-19, topic=__consumer_offsets, partition=19, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 442ms (24/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:31,371 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:31,371 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:31,965 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,028 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-2, topic=__consumer_offsets, partition=2, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 747ms (25/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:32,113 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,114 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,352 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,407 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-20, topic=__consumer_offsets, partition=20, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 379ms (26/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:32,496 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,496 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,819 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:32,898 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-21, topic=__consumer_offsets, partition=21, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 485ms (27/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:33,013 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,013 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,399 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,471 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-22, topic=__consumer_offsets, partition=22, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 573ms (28/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:33,600 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,601 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,842 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,905 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-23, topic=__consumer_offsets, partition=23, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 434ms (29/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:33,999 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:33,999 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:34,372 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:34,430 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-24, topic=__consumer_offsets, partition=24, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 525ms (30/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:34,512 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:34,512 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:34,815 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:34,882 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-25, topic=__consumer_offsets, partition=25, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 452ms (31/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:34,974 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:34,974 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:35,238 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:35,302 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-26, topic=__consumer_offsets, partition=26, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 419ms (32/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:35,391 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:35,392 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:35,655 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:35,721 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-27, topic=__consumer_offsets, partition=27, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 418ms (33/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:35,811 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:35,811 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,057 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,116 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-28, topic=__consumer_offsets, partition=28, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 395ms (34/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:36,199 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,199 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,434 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,492 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-29, topic=__consumer_offsets, partition=29, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 375ms (35/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:36,570 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,571 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:36,971 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,029 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-3, topic=__consumer_offsets, partition=3, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 538ms (36/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:37,109 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,109 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,344 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,413 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-30, topic=__consumer_offsets, partition=30, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 383ms (37/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:37,503 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,504 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,812 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,874 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-31, topic=__consumer_offsets, partition=31, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 460ms (38/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:37,967 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:37,967 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:38,261 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:38,324 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-32, topic=__consumer_offsets, partition=32, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 451ms (39/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:38,401 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:38,401 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:38,619 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:38,681 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-33, topic=__consumer_offsets, partition=33, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 356ms (40/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:38,776 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:38,776 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,003 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,065 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-34, topic=__consumer_offsets, partition=34, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 384ms (41/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:39,153 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,153 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,383 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,439 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-35, topic=__consumer_offsets, partition=35, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 374ms (42/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:39,533 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,534 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,834 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:39,910 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-36, topic=__consumer_offsets, partition=36, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 471ms (43/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:40,010 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:40,011 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:40,291 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:40,358 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-37, topic=__consumer_offsets, partition=37, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 447ms (44/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:40,455 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:40,456 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:40,911 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:40,971 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-38, topic=__consumer_offsets, partition=38, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 613ms (45/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:41,051 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,051 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,303 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,364 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-39, topic=__consumer_offsets, partition=39, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 392ms (46/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:41,442 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,443 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,683 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,757 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-4, topic=__consumer_offsets, partition=4, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 392ms (47/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:41,854 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:41,854 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:42,168 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:42,235 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-40, topic=__consumer_offsets, partition=40, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 478ms (48/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:42,318 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:42,318 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:42,589 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:42,659 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-41, topic=__consumer_offsets, partition=41, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 423ms (49/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:42,877 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:42,878 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:43,106 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:43,165 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-42, topic=__consumer_offsets, partition=42, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 505ms (50/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:43,247 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:43,247 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:43,477 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:43,533 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-43, topic=__consumer_offsets, partition=43, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 368ms (51/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:43,702 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:43,702 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:44,011 INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 3 (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:44,040 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 3 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:44,040 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 3 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:44,090 INFO [ProducerStateManager partition=__consumer_offsets-44] Loading producer state from snapshot file '/var/lib/kafka/data/kafka-log0/__consumer_offsets-44/00000000000000000003.snapshot' (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:44,258 INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 4 (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:44,693 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 4 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:44,746 INFO [ProducerStateManager partition=__consumer_offsets-44] Loading producer state from snapshot file '/var/lib/kafka/data/kafka-log0/__consumer_offsets-44/00000000000000000004.snapshot' (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:45,311 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-44, topic=__consumer_offsets, partition=44, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=4) with 2 segments in 1778ms (52/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:45,419 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:45,419 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:45,753 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:45,821 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-45, topic=__consumer_offsets, partition=45, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 509ms (53/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:45,905 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:45,905 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:46,298 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:46,370 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-46, topic=__consumer_offsets, partition=46, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 550ms (54/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:46,456 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:46,456 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:46,712 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:46,770 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-47, topic=__consumer_offsets, partition=47, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 399ms (55/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:46,852 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:46,852 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:47,270 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:47,332 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-48, topic=__consumer_offsets, partition=48, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 562ms (56/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:47,423 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:47,423 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:47,652 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:47,709 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-49, topic=__consumer_offsets, partition=49, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 376ms (57/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:47,792 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:47,792 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:48,258 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:48,319 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-5, topic=__consumer_offsets, partition=5, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 609ms (58/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:48,415 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:48,415 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:48,679 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:48,734 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-6, topic=__consumer_offsets, partition=6, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 414ms (59/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:48,808 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:48,808 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,055 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,114 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-7, topic=__consumer_offsets, partition=7, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 379ms (60/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:49,200 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,200 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,454 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,519 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-8, topic=__consumer_offsets, partition=8, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 405ms (61/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:49,595 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,596 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,825 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,888 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__consumer_offsets-9, topic=__consumer_offsets, partition=9, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 368ms (62/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:49,972 INFO [Log partition=__strimzi_store_topic-0, dir=/var/lib/kafka/data/kafka-log0] Recovering unflushed segment 7 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:49,972 INFO [Log partition=__strimzi_store_topic-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 7 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:50,031 INFO [ProducerStateManager partition=__strimzi_store_topic-0] Writing producer snapshot at offset 7 (kafka.log.ProducerStateManager) [pool-4-thread-1] 2021-03-29 17:25:50,249 INFO [Log partition=__strimzi_store_topic-0, dir=/var/lib/kafka/data/kafka-log0] Loading producer state till offset 7 with message format version 2 (kafka.log.Log) [pool-4-thread-1] 2021-03-29 17:25:50,325 INFO Completed load of Log(dir=/var/lib/kafka/data/kafka-log0/__strimzi_store_topic-0, topic=__strimzi_store_topic, partition=0, highWatermark=7, lastStableOffset=7, logStartOffset=7, logEndOffset=7) with 1 segments in 438ms (63/63 loaded in /var/lib/kafka/data/kafka-log0) (kafka.log.LogManager) [pool-4-thread-1] 2021-03-29 17:25:50,334 INFO Loaded 63 logs in 35545ms. (kafka.log.LogManager) [main] 2021-03-29 17:25:50,366 INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [main] 2021-03-29 17:25:50,367 INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [main] 2021-03-29 17:25:50,370 INFO Starting the log cleaner (kafka.log.LogCleaner) [main] 2021-03-29 17:25:50,428 INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,507 INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-44. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,508 INFO Cleaner 0: Building offset map for __consumer_offsets-44... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,545 INFO Cleaner 0: Building offset map for log __consumer_offsets-44 for 1 segments in offset range [0, 3). (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,579 INFO Cleaner 0: Offset map for log __consumer_offsets-44 complete. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,617 INFO Cleaner 0: Cleaning log __consumer_offsets-44 (cleaning prior to Thu Mar 18 12:40:57 UTC 2021, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,666 INFO Cleaner 0: Cleaning LogSegment(baseOffset=0, size=1036, lastModifiedTime=1616071257467, largestRecordTimestamp=Some(1616071225581)) in log __consumer_offsets-44 into 0 with deletion horizon 0, retaining deletes. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:50,944 INFO Created ConnectionAcceptRate sensor, quotaLimit=2147483647 (kafka.network.ConnectionQuotas) [main] 2021-03-29 17:25:50,946 INFO Created ConnectionAcceptRate-REPLICATION-9091 sensor, quotaLimit=2147483647 (kafka.network.ConnectionQuotas) [main] 2021-03-29 17:25:50,951 INFO Updated REPLICATION-9091 max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [main] 2021-03-29 17:25:50,955 INFO Awaiting socket connections on 0.0.0.0:9091. (kafka.network.Acceptor) [main] 2021-03-29 17:25:51,070 ERROR Failed to clean up log for __consumer_offsets-44 in dir /var/lib/kafka/data/kafka-log0 due to IOException (kafka.server.LogDirFailureChannel) [kafka-log-cleaner-thread-0] java.nio.file.FileSystemException: /var/lib/kafka/data/kafka-log0/__consumer_offsets-44/00000000000000000000.log.cleaned: Operation not permitted at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.setTimes(UnixFileAttributeViews.java:125) at java.base/java.nio.file.Files.setLastModifiedTime(Files.java:2355) at kafka.log.LogSegment.lastModified_$eq(LogSegment.scala:651) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:610) at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:539) at kafka.log.Cleaner.doClean(LogCleaner.scala:538) at kafka.log.Cleaner.clean(LogCleaner.scala:512) at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:381) at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:353) at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:333) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:322) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) 2021-03-29 17:25:51,309 INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-44. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:51,309 INFO Cleaner 0: Building offset map for __consumer_offsets-44... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:52,032 INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : ListenerName(REPLICATION-9091) (kafka.network.SocketServer) [main] 2021-03-29 17:25:52,033 INFO Created ConnectionAcceptRate-PLAIN-9092 sensor, quotaLimit=2147483647 (kafka.network.ConnectionQuotas) [main] 2021-03-29 17:25:52,034 INFO Updated PLAIN-9092 max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [main] 2021-03-29 17:25:52,035 INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor) [main] 2021-03-29 17:25:52,051 INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : ListenerName(PLAIN-9092) (kafka.network.SocketServer) [main] 2021-03-29 17:25:52,122 INFO Cleaner 0: Building offset map for log __consumer_offsets-44 for 1 segments in offset range [0, 3). (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:52,123 INFO Cleaner 0: Offset map for log __consumer_offsets-44 complete. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:52,136 INFO Cleaner 0: Cleaning log __consumer_offsets-44 (cleaning prior to Thu Mar 18 12:40:57 UTC 2021, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:52,258 INFO Cleaner 0: Cleaning LogSegment(baseOffset=0, size=1036, lastModifiedTime=1616071257467, largestRecordTimestamp=Some(1616071225581)) in log __consumer_offsets-44 into 0 with deletion horizon 0, retaining deletes. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0] 2021-03-29 17:25:52,301 INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Produce] 2021-03-29 17:25:52,316 INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-DeleteRecords] 2021-03-29 17:25:52,316 INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Fetch] 2021-03-29 17:25:52,321 INFO [ExpirationReaper-0-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-ElectLeader] 2021-03-29 17:25:52,363 INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler) [LogDirFailureHandler] 2021-03-29 17:25:52,367 WARN [ReplicaManager broker=0] Stopping serving replicas in dir /var/lib/kafka/data/kafka-log0 (kafka.server.ReplicaManager) [LogDirFailureHandler] 2021-03-29 17:25:52,367 INFO [broker-0-to-controller-send-thread]: Starting (kafka.server.BrokerToControllerRequestThread) [broker-0-to-controller-send-thread] 2021-03-29 17:25:52,380 WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions and stopped moving logs for partitions because they are in the failed log directory /var/lib/kafka/data/kafka-log0. (kafka.server.ReplicaManager) [LogDirFailureHandler] 2021-03-29 17:25:52,381 WARN Stopping serving logs in dir /var/lib/kafka/data/kafka-log0 (kafka.log.LogManager) [LogDirFailureHandler] 2021-03-29 17:25:52,385 ERROR Shutdown broker because all log dirs in /var/lib/kafka/data/kafka-log0 have failed (kafka.log.LogManager) [LogDirFailureHandler]

Logs from the crashing zookeeper pod 'Detected Zookeeper ID 2 Preparing truststore Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca Certificate was added to keystore Preparing truststore is complete Looking for the right CA Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt Preparing keystore for client and quorum listeners Preparing keystore for client and quorum listeners is complete Starting Zookeeper with configuration:

The directory where the snapshot is stored.

dataDir=/var/lib/zookeeper/data

Other options

4lw.commands.whitelist=* standaloneEnabled=false reconfigEnabled=true clientPort=12181 clientPortAddress=127.0.0.1

TLS options

serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.clientAuth=need ssl.quorum.clientAuth=need secureClientPort=2181 sslQuorum=true

ssl.trustStore.location=/tmp/zookeeper/cluster.truststore.p12 ssl.trustStore.password=DbXXvQUExpHceIqkGXdKusL7dxkQSNbS ssl.trustStore.type=PKCS12 ssl.quorum.trustStore.location=/tmp/zookeeper/cluster.truststore.p12 ssl.quorum.trustStore.password=DbXXvQUExpHceIqkGXdKusL7dxkQSNbS ssl.quorum.trustStore.type=PKCS12

ssl.keyStore.location=/tmp/zookeeper/cluster.keystore.p12 ssl.keyStore.password=DbXXvQUExpHceIqkGXdKusL7dxkQSNbS ssl.keyStore.type=PKCS12 ssl.quorum.keyStore.location=/tmp/zookeeper/cluster.keystore.p12 ssl.quorum.keyStore.password=DbXXvQUExpHceIqkGXdKusL7dxkQSNbS ssl.quorum.keyStore.type=PKCS12

Provided configuration

tickTime=2000 initLimit=5 syncLimit=2 autopurge.purgeInterval=1

Zookeeper nodes configuration

server.1=kafka-zookeeper-0.kafka-zookeeper-nodes.svc:2888:3888:participant;127.0.0.1:12181 server.2=kafka-zookeeper-1.kafka-zookeeper-nodes.svc:2888:3888:participant;127.0.0.1:12181 server.3=kafka-zookeeper-2.kafka-zookeeper-nodes.svc:2888:3888:participant;127.0.0.1:12181

2021-03-29 17:30:41,277 INFO Reading configuration from: /tmp/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] 2021-03-29 17:30:41,285 INFO clientPortAddress is 127.0.0.1:12181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] 2021-03-29 17:30:41,286 INFO secureClientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] 2021-03-29 17:30:41,289 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main] 2021-03-29 17:30:41,367 INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [main] 2021-03-29 17:30:41,368 INFO autopurge.purgeInterval set to 1 (org.apache.zookeeper.server.DatadirCleanupManager) [main] 2021-03-29 17:30:41,370 INFO Purge task started. (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask] 2021-03-29 17:30:41,382 INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [PurgeTask] 2021-03-29 17:30:41,383 INFO Log4j 1.2 jmx support found and enabled. (org.apache.zookeeper.jmx.ManagedUtil) [main] 2021-03-29 17:30:41,398 INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain) [main] 2021-03-29 17:30:41,447 INFO Purge task completed. (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask] 2021-03-29 17:30:41,483 INFO zookeeper.client.portUnification=false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] 2021-03-29 17:30:41,538 INFO Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [main] 2021-03-29 17:30:41,540 INFO zookeeper.client.portUnification=false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] 2021-03-29 17:30:41,541 INFO Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [main] 2021-03-29 17:30:41,565 INFO Logging initialized @920ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [main] 2021-03-29 17:30:41,662 WARN o.e.j.s.ServletContextHandler@2d710f1a{/,null,STOPPED} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler) [main] 2021-03-29 17:30:41,662 WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler) [main] 2021-03-29 17:30:41,682 INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [main] 2021-03-29 17:30:41,682 INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,682 INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,682 INFO tickTime set to 2000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,682 INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,682 INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,682 INFO initLimit set to 5 (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,694 INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [main] 2021-03-29 17:30:41,696 INFO Using TLS encrypted quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,696 INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,696 INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,696 INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [main] 2021-03-29 17:30:41,715 INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap) [main] 2021-03-29 17:30:41,813 ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer) [main] java.io.IOException: Unreasonable length = 1911907914 at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:146) at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:111) at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:205) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:684) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:294) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:229) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:253) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) 2021-03-29 17:30:41,816 ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain) [main] java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:938) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) Caused by: java.io.IOException: Unreasonable length = 1911907914 at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:146) at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:111) at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:205) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:684) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:294) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:229) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:253) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901) ... 4 more'

If you need more info do not hesitate to ask.

Best, Marc

scholzj commented 3 years ago

That looks like some storage issue - AFAIK Azure File is NFS-like storage and I'm not sure it is expected to work. The log snippets also don't make it clear where they are coming from and how do they fit in - please always provide full logs.

msts1906 commented 3 years ago

That looks like some storage issue - AFAIK Azure File is NFS-like storage and I'm not sure it is expected to work. The log snippets also don't make it clear where they are coming from and how do they fit in - please always provide full logs.

We want to operate the cluster in cloud environment, what other choice then using NFS like storage do we have?

scholzj commented 3 years ago

Interesting. For me, NFS is more or less the synonym for the world before cloud. I guess different backgrounds do that :-).

Anyway, I think that on Azure, Azure Disk is block storage which works well (AWS EBS volumes and GCP Persistent Disks would be the options in other public clouds). But I do not have that much experience with Azure, so I'm not saying on 100% this is the issue.

msts1906 commented 3 years ago

I changed the yaml to use azure disk and applied it, yet nothing happened. Is there a way to restart the cluster?

scholzj commented 3 years ago

I don't think that is normally supported with different storage types. Maybe Azure allows you do some snapshots and start new volumes from the snapshots? Or you can start a new cluster and mirror the data to it / or copy them manually.

msts1906 commented 3 years ago

Since we didn't had any data in the cluster yet (we are in early dev stages), we just uninstalled strimzi and reinstalled it after we upgraded to Kubernetes version 1.20.2. It works for now, yet it is unsatisfying, that any change to the cluster leads to strimzi not functioning properly

scholzj commented 3 years ago

Well, if it is unsatisfying then maybe you should have invested more time into investigating what the issue actually was. You cannot expect someone else to do things for you out of 2 exceptions and without sharing full logs etc.