search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.86k stars 1.3k forks source link

Cluster operator does not have sufficient memory: it uses up >98% of memory #4471

Closed fzyzcjy closed 2 years ago

fzyzcjy commented 3 years ago

Describe the bug The memory is not sufficient for strimzi-cluster-operator after running two weeks. It is even causing alerts!

image

This is the strimzi-cluster-operator's memory usage:

image

Notice it is even using Swap space now:

image

cross-validating the memory usage:

image

To Reproduce Steps to reproduce the behavior:

  1. Run strimzi
  2. wait for two weeks
  3. see the memory problem

Expected behavior no mem problem

Environment (please complete the following information):

YAML files and logs N/A

Additional context N/A

sknot-rh commented 3 years ago

How many resources does Cluster Operator manage?

fzyzcjy commented 3 years ago

@sknot-rh only one Kafka with a few topics. Following are CRD resources:

image image

scholzj commented 3 years ago

I'm not sure I necessarily understand the problem. The memory is there to be used so it is used. Do you see any problems for example with JVM garbage collection? For example taking too much time etc.?

fzyzcjy commented 3 years ago

@scholzj Netdata (a monitoring system) complains that it uses 98% of memory, so it is dangerous and going to be killed. And it also starts using some swapping.

Maybe this is not a problem (e.g. because it will never actually get OOM?)

EDIT

Oh I do see it get killed by OOM:

image

It was terminated (and later restarted) a week ago. And this exit code 137 seems to the code for OOM.

So IMHO this is a problem - of course we do not want it to be killed by OOM!

scholzj commented 3 years ago

Well, the application does not control use of swap. I think that normally it is recommended to not have any swap for Kubernetes. But I guess not much we can do about it.

fzyzcjy commented 3 years ago

@scholzj I also think we should not have swap. That may be opened by the Docker image of Strimzi.

The biggest problem is, this is even get killed because of OOM. So this is a kind of bug :(

scholzj commented 3 years ago

I do not think container can configure your worker node and enable swap. That is your system settings.

fzyzcjy commented 3 years ago

@scholzj Hmm I will check it. Anyway my host machine has zero swap:

image

By the way, the problem is worse without swap: It will be OOM kiled.

scholzj commented 3 years ago

Then how can the container swap if you have no swap? Maybe it is just a generic message from your monitoring which is printed regardless whether actual swap is or isn't used?

As for the OOM, you can probably share your logs if your want to see what the operator is doing. You can also try to fine-tune the memory - play with the resource configuration or with the Java -Xmx setting.

fzyzcjy commented 3 years ago

Logs

2021-02-25 10:55:04 INFO  AbstractOperator:318 - Reconciliation #4260(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 10:57:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 10:57:01 INFO  AbstractOperator:173 - Reconciliation #4261(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 10:57:04 INFO  AbstractOperator:318 - Reconciliation #4261(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 10:59:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 10:59:01 INFO  AbstractOperator:173 - Reconciliation #4262(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 10:59:04 INFO  AbstractOperator:318 - Reconciliation #4262(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:01:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:01:01 INFO  AbstractOperator:173 - Reconciliation #4263(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:01:04 INFO  AbstractOperator:318 - Reconciliation #4263(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:03:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:03:01 INFO  AbstractOperator:173 - Reconciliation #4264(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:03:04 INFO  AbstractOperator:318 - Reconciliation #4264(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:05:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:05:01 INFO  AbstractOperator:173 - Reconciliation #4265(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:05:04 INFO  AbstractOperator:318 - Reconciliation #4265(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:07:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:07:01 INFO  AbstractOperator:173 - Reconciliation #4266(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:07:04 INFO  AbstractOperator:318 - Reconciliation #4266(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:09:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:09:01 INFO  AbstractOperator:173 - Reconciliation #4267(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:09:04 INFO  AbstractOperator:318 - Reconciliation #4267(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:11:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:11:01 INFO  AbstractOperator:173 - Reconciliation #4268(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:11:04 INFO  AbstractOperator:318 - Reconciliation #4268(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:13:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:13:01 INFO  AbstractOperator:173 - Reconciliation #4269(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:13:05 INFO  AbstractOperator:318 - Reconciliation #4269(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:15:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:15:01 INFO  AbstractOperator:173 - Reconciliation #4270(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:15:05 INFO  AbstractOperator:318 - Reconciliation #4270(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:17:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:17:01 INFO  AbstractOperator:173 - Reconciliation #4271(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:17:05 INFO  AbstractOperator:318 - Reconciliation #4271(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:19:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:19:01 INFO  AbstractOperator:173 - Reconciliation #4272(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:19:05 INFO  AbstractOperator:318 - Reconciliation #4272(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:21:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:21:01 INFO  AbstractOperator:173 - Reconciliation #4273(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:21:05 INFO  AbstractOperator:318 - Reconciliation #4273(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:23:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:23:01 INFO  AbstractOperator:173 - Reconciliation #4274(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:23:05 INFO  AbstractOperator:318 - Reconciliation #4274(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:25:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:25:01 INFO  AbstractOperator:173 - Reconciliation #4275(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:25:04 INFO  AbstractOperator:318 - Reconciliation #4275(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:27:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:27:01 INFO  AbstractOperator:173 - Reconciliation #4276(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:27:04 INFO  AbstractOperator:318 - Reconciliation #4276(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:29:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:29:01 INFO  AbstractOperator:173 - Reconciliation #4277(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:29:04 INFO  AbstractOperator:318 - Reconciliation #4277(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:31:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:31:01 INFO  AbstractOperator:173 - Reconciliation #4278(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:31:04 INFO  AbstractOperator:318 - Reconciliation #4278(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:33:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:33:01 INFO  AbstractOperator:173 - Reconciliation #4279(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:33:04 INFO  AbstractOperator:318 - Reconciliation #4279(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:35:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:35:01 INFO  AbstractOperator:173 - Reconciliation #4280(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:35:04 INFO  AbstractOperator:318 - Reconciliation #4280(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:37:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:37:01 INFO  AbstractOperator:173 - Reconciliation #4281(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:37:05 INFO  AbstractOperator:318 - Reconciliation #4281(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:39:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:39:01 INFO  AbstractOperator:173 - Reconciliation #4282(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:39:05 INFO  AbstractOperator:318 - Reconciliation #4282(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:41:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:41:01 INFO  AbstractOperator:173 - Reconciliation #4283(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:41:05 INFO  AbstractOperator:318 - Reconciliation #4283(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:43:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:43:01 INFO  AbstractOperator:173 - Reconciliation #4284(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:43:04 INFO  AbstractOperator:318 - Reconciliation #4284(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:45:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:45:01 INFO  AbstractOperator:173 - Reconciliation #4285(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:45:10 INFO  AbstractOperator:318 - Reconciliation #4285(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:47:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:47:01 INFO  AbstractOperator:173 - Reconciliation #4286(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:47:07 INFO  AbstractOperator:318 - Reconciliation #4286(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:49:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:49:01 INFO  AbstractOperator:173 - Reconciliation #4287(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:49:10 INFO  AbstractOperator:318 - Reconciliation #4287(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:51:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:51:10 INFO  AbstractOperator:173 - Reconciliation #4288(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:51:35 INFO  AbstractOperator:318 - Reconciliation #4288(timer) Kafka(default/tom-kafka): reconciled
Feb 25, 2021 11:51:04 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2434 ms, time limit is 2000 ms
2021-02-25 11:53:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:53:01 INFO  AbstractOperator:173 - Reconciliation #4289(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:53:15 INFO  AbstractOperator:318 - Reconciliation #4289(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:55:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:55:01 INFO  AbstractOperator:173 - Reconciliation #4290(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:55:11 INFO  AbstractOperator:318 - Reconciliation #4290(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:57:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 11:57:01 INFO  AbstractOperator:173 - Reconciliation #4291(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:57:09 INFO  AbstractOperator:318 - Reconciliation #4291(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 11:59:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
Feb 25, 2021 11:59:04 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2845 ms, time limit is 2000 ms
Feb 25, 2021 11:59:04 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 3851 ms, time limit is 2000 ms
Feb 25, 2021 11:59:05 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 4858 ms, time limit is 2000 ms
Feb 25, 2021 11:59:07 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 5864 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:150)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 11:59:08 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 6871 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:150)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 11:59:09 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 7900 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base@11.0.7/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:584)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 11:59:10 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 8908 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

2021-02-25 11:59:10 INFO  AbstractOperator:173 - Reconciliation #4292(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 11:59:17 INFO  AbstractOperator:318 - Reconciliation #4292(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:01:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:01:01 INFO  AbstractOperator:173 - Reconciliation #4293(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:01:09 INFO  AbstractOperator:318 - Reconciliation #4293(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:03:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:03:01 INFO  AbstractOperator:173 - Reconciliation #4294(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:03:09 INFO  AbstractOperator:318 - Reconciliation #4294(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:05:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:05:01 INFO  AbstractOperator:173 - Reconciliation #4295(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:05:08 INFO  AbstractOperator:318 - Reconciliation #4295(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:07:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:07:01 INFO  AbstractOperator:173 - Reconciliation #4296(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:07:08 INFO  AbstractOperator:318 - Reconciliation #4296(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:09:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:09:01 INFO  AbstractOperator:173 - Reconciliation #4297(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:09:07 INFO  AbstractOperator:318 - Reconciliation #4297(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:11:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:11:01 INFO  AbstractOperator:173 - Reconciliation #4298(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:11:07 INFO  AbstractOperator:318 - Reconciliation #4298(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:13:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-25 12:13:01 INFO  AbstractOperator:173 - Reconciliation #4299(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-25 12:13:05 INFO  AbstractOperator:318 - Reconciliation #4299(timer) Kafka(default/tom-kafka): reconciled
2021-02-25 12:15:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
Feb 25, 2021 12:15:03 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2444 ms, time limit is 2000 ms
Feb 25, 2021 12:15:04 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 3457 ms, time limit is 2000 ms
Feb 25, 2021 12:15:05 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 4492 ms, time limit is 2000 ms
Feb 25, 2021 12:15:06 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 5504 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:07 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 6512 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base@11.0.7/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base@11.0.7/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:556)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:08 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 7528 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:09 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 8558 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base@11.0.7/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base@11.0.7/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:556)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:10 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 9597 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.initClassName(Native Method)
    at java.base@11.0.7/java.lang.Class.getName(Class.java:793)
    at java.base@11.0.7/java.lang.Class.getPackageName(Class.java:1008)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyModuleAccess(Reflection.java:211)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyMemberAccess(Reflection.java:125)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.ensureMemberAccess(Reflection.java:98)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:579)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:11 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 10615 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at jdk.internal.reflect.GeneratedMethodAccessor1240.<init>(Unknown Source)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base@11.0.7/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:584)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:12 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 11635 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:13 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 12672 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base@11.0.7/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base@11.0.7/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:556)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:14 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 13687 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at jdk.internal.reflect.GeneratedMethodAccessor1251.<init>(Unknown Source)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base@11.0.7/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:584)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:15 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 14696 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:16 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 15705 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at jdk.internal.reflect.GeneratedMethodAccessor1263.<init>(Unknown Source)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base@11.0.7/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:584)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:17 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 16724 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:18 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 17734 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.String.intern(Native Method)
    at java.base@11.0.7/java.lang.Class.getPackageName(Class.java:1010)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyModuleAccess(Reflection.java:211)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyMemberAccess(Reflection.java:125)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.ensureMemberAccess(Reflection.java:98)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:579)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:20 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 18748 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:20 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 19777 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.String.intern(Native Method)
    at java.base@11.0.7/java.lang.Class.getPackageName(Class.java:1010)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyModuleAccess(Reflection.java:211)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyMemberAccess(Reflection.java:125)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.ensureMemberAccess(Reflection.java:98)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:579)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:22 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 20790 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:23 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 21800 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:24 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 22821 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:25 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 23834 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:26 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 24843 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.util.Hashtable.<init>(Hashtable.java:185)
    at java.base@11.0.7/java.util.Hashtable.<init>(Hashtable.java:216)
    at java.base@11.0.7/java.lang.ClassLoader.<init>(ClassLoader.java:386)
    at java.base@11.0.7/java.lang.ClassLoader.<init>(ClassLoader.java:457)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingClassLoader.<init>(ClassDefiner.java:72)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner$1.run(ClassDefiner.java:60)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner$1.run(ClassDefiner.java:58)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:57)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:27 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 25853 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base@11.0.7/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base@11.0.7/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:556)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:28 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 26862 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at jdk.internal.reflect.GeneratedMethodAccessor1314.<init>(Unknown Source)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base@11.0.7/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:584)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:29 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 27928 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base@11.0.7/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base@11.0.7/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:556)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:30 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 28943 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/java.lang.Class.initClassName(Native Method)
    at java.base@11.0.7/java.lang.Class.getName(Class.java:793)
    at java.base@11.0.7/java.lang.Class.getPackageName(Class.java:1008)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyModuleAccess(Reflection.java:211)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.verifyMemberAccess(Reflection.java:125)
    at java.base@11.0.7/jdk.internal.reflect.Reflection.ensureMemberAccess(Reflection.java:98)
    at java.base@11.0.7/java.lang.Class.newInstance(Class.java:579)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:404)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:31 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 29952 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:32 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 30984 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:33 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 32055 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:34 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 33069 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:35 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 34082 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)

Feb 25, 2021 12:15:36 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 35091 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass0(Native Method)
    at java.base@11.0.7/jdk.internal.misc.Unsafe.defineClass(Unsafe.java:1192)
    at java.base@11.0.7/jdk.internal.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:400)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
    at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
    at java.base@11.0.7/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
    at java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
    at java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.7/java.lang.reflect.Method.invoke(Method.java:566)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:109)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visitProperty(ResourceVisitor.java:157)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:111)
    at app//io.strimzi.operator.common.model.ResourceVisitor.visit(ResourceVisitor.java:74)
    at app//io.strimzi.operator.common.AbstractOperator.validate(AbstractOperator.java:263)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$reconcile$3(AbstractOperator.java:172)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$236/0x00000008403bb040.call(Unknown Source)
    at app//io.strimzi.operator.common.AbstractOperator.lambda$withLock$6(AbstractOperator.java:230)
    at app//io.strimzi.operator.common.AbstractOperator$$Lambda$237/0x00000008403c3c40.handle(Unknown Source)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter.lambda$acquireLock$1(LocalAsyncLocks.java:65)
    at app//io.vertx.core.shareddata.impl.LocalAsyncLocks$LockWaiter$$Lambda$240/0x00000008403c5440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$201/0x0000000840318040.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)
fzyzcjy commented 3 years ago

Hmm I see sth interesting: what is that exception? (maybe caused by the nearly full memory?)

sknot-rh commented 3 years ago

Can you share the Kafka resource (yaml)?

scholzj commented 3 years ago

Hmm I see sth interesting: what is that exception? (maybe caused by the nearly full memory?)

That normally indicates that you do not have enough CPU but I guess it could be also caused by garbage collection. How did you installed the operator? Did you do any changes to the default resource settings? Also, do you have Prometheus monitoring? That shows the operator metrics about memory, GC etc. which would be interesting.

sushiMix commented 3 years ago

Hello I have a similar issue with an OOM killed operator on my kubernetes cluster. I'm using the helm chart to install the operator and I don't override any resource management by default

For this test I increased the memory limit to 512M just to check if it was too low or is there a constant increase.

Here is the profile of the memory used (container memory usage).

image 
  lastState:
    terminated:
      exitCode: 137
      reason: OOMKilled
      startedAt: '2021-02-18T08:32:18Z'
      finishedAt: '2021-02-21T08:38:24Z'

For now : we only noticed error on kubecontroller for PVC update linked to "strimzi world" (endless retry ?) E0225 15:28:28.715190 1 pvc_protection_controller.go:140] PVC default/data-kafka-kafka-0 failed with : Operation cannot be fulfilled on persistentvolumeclaims "data-kafka-kafka-0": the object has been modified; please apply your changes to the latest version and try again

I'm using 0.20.0 operator and tried 0.21.0 on kubernetes 1.18 I have 1 kafka + zookeeper + bridge

scholzj commented 3 years ago

E0225 15:28:28.715190 1 pvc_protection_controller.go:140] PVC default/data-kafka-kafka-0 failed with : Operation cannot be fulfilled on persistentvolumeclaims "data-kafka-kafka-0": the object has been modified; please apply your changes to the latest version and try again

This is a general Kube message. It is probably related to the PVC being managed by the operator.

fzyzcjy commented 3 years ago

Can you share the Kafka resource (yaml)?

@sknot-rh

Sure. here is the tom-kafka Kafka resource on my running cluster:

kind: Kafka
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  annotations:
    meta.helm.sh/release-name: tom-kafka
    meta.helm.sh/release-namespace: default
  creationTimestamp: '2020-11-09T13:41:10Z'
  generation: 9
  labels:
    app.kubernetes.io/instance: tom-kafka
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: tom-kafka
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: tom-kafka-0.1.0
  name: tom-kafka
  namespace: default
  resourceVersion: '41665489'
  selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/default/kafkas/tom-kafka
  uid: e0de35ea-df44-4827-8568-6e3eaf1e7d34
spec:
  entityOperator:
    template:
      pod:
        metadata:
          annotations:
            tom_filebeat_mode: kafka_entity_operator
    topicOperator:
      resources:
        limits:
          cpu: 4000m
          memory: 4096Mi
        requests:
          cpu: 25m
          memory: 32Mi
    userOperator:
      resources:
        limits:
          cpu: 4000m
          memory: 4096Mi
        requests:
          cpu: 25m
          memory: 32Mi
  kafka:
    config:
      log.message.format.version: '2.5'
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
      plain: {}
      tls: {}
    replicas: 1
    resources:
      limits:
        cpu: 4000m
        memory: 1024Mi
      requests:
        cpu: 25m
        memory: 768Mi
    storage:
      class: csi-disk
      deleteClaim: false
      size: 8Gi
      type: persistent-claim
    template:
      persistentVolumeClaim:
        metadata:
          annotations:
            everest.io/disk-volume-type: SAS
          labels:
            failure-domain.beta.kubernetes.io/region: cn-south-1
            failure-domain.beta.kubernetes.io/zone: cn-south-1c
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: tom-kafka
                      app.kubernetes.io/name: tom-kafka
                      role: kafka
                  topologyKey: kubernetes.io/hostname
                weight: 100
        imagePullSecrets:
          - name: default-secret
        metadata:
          annotations:
            tom_filebeat_mode: kafka
          labels:
            role: kafka
    version: 2.5.0
  kafkaExporter:
    enableSaramaLogging: true
    groupRegex: .*
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    logging: debug
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      limits:
        cpu: 4000m
        memory: 1024Mi
      requests:
        cpu: 200m
        memory: 64Mi
    template:
      pod:
        imagePullSecrets:
          - name: default-secret
        metadata:
          annotations:
            prometheus.io/port: '9404'
            prometheus.io/scrape: 'true'
        securityContext:
          fsGroup: 0
          runAsUser: 1000001
        terminationGracePeriodSeconds: 120
    topicRegex: .*
  zookeeper:
    replicas: 1
    resources:
      limits:
        cpu: 4000m
        memory: 768Mi
      requests:
        cpu: 25m
        memory: 300Mi
    storage:
      class: csi-disk
      deleteClaim: false
      size: 8Gi
      type: persistent-claim
    template:
      persistentVolumeClaim:
        metadata:
          annotations:
            everest.io/disk-volume-type: SAS
          labels:
            failure-domain.beta.kubernetes.io/region: cn-south-1
            failure-domain.beta.kubernetes.io/zone: cn-south-1c
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: tom-kafka
                      app.kubernetes.io/name: tom-kafka
                      role: zookeeper
                  topologyKey: kubernetes.io/hostname
                weight: 100
        imagePullSecrets:
          - name: default-secret
        metadata:
          annotations:
            tom_filebeat_mode: zookeeper
          labels:
            role: zookeeper
status:
  conditions:
    - lastTransitionTime: '2021-02-16T07:09:41+0000'
      status: 'True'
      type: Ready
  listeners:
    - addresses:
        - host: tom-kafka-kafka-bootstrap.default.svc
          port: 9092
      bootstrapServers: 'tom-kafka-kafka-bootstrap.default.svc:9092'
      type: plain
    - addresses:
        - host: tom-kafka-kafka-bootstrap.default.svc
          port: 9093
      bootstrapServers: 'tom-kafka-kafka-bootstrap.default.svc:9093'
      certificates:
        - |
          -----BEGIN CERTIFICATE-----
            ...
          -----END CERTIFICATE-----
      type: tls
  observedGeneration: 9
fzyzcjy commented 3 years ago

@scholzj

Actually when I look at it now, it has no more errors:

2021-02-26 06:43:06 INFO  AbstractOperator:318 - Reconciliation #4854(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:45:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:45:01 INFO  AbstractOperator:173 - Reconciliation #4855(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:45:07 INFO  AbstractOperator:318 - Reconciliation #4855(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:47:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:47:01 INFO  AbstractOperator:173 - Reconciliation #4856(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:47:05 INFO  AbstractOperator:318 - Reconciliation #4856(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:49:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:49:01 INFO  AbstractOperator:173 - Reconciliation #4857(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:49:04 INFO  AbstractOperator:318 - Reconciliation #4857(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:51:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:51:01 INFO  AbstractOperator:173 - Reconciliation #4858(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:51:05 INFO  AbstractOperator:318 - Reconciliation #4858(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:53:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:53:01 INFO  AbstractOperator:173 - Reconciliation #4859(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:53:05 INFO  AbstractOperator:318 - Reconciliation #4859(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:55:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:55:01 INFO  AbstractOperator:173 - Reconciliation #4860(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:55:05 INFO  AbstractOperator:318 - Reconciliation #4860(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:57:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:57:01 INFO  AbstractOperator:173 - Reconciliation #4861(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:57:07 INFO  AbstractOperator:318 - Reconciliation #4861(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 06:59:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 06:59:01 INFO  AbstractOperator:173 - Reconciliation #4862(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 06:59:06 INFO  AbstractOperator:318 - Reconciliation #4862(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:01:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:01:01 INFO  AbstractOperator:173 - Reconciliation #4863(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:01:06 INFO  AbstractOperator:318 - Reconciliation #4863(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:03:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:03:01 INFO  AbstractOperator:173 - Reconciliation #4864(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:03:06 INFO  AbstractOperator:318 - Reconciliation #4864(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:05:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:05:01 INFO  AbstractOperator:173 - Reconciliation #4865(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:05:05 INFO  AbstractOperator:318 - Reconciliation #4865(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:07:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:07:01 INFO  AbstractOperator:173 - Reconciliation #4866(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:07:05 INFO  AbstractOperator:318 - Reconciliation #4866(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:09:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:09:01 INFO  AbstractOperator:173 - Reconciliation #4867(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:09:05 INFO  AbstractOperator:318 - Reconciliation #4867(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:11:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:11:01 INFO  AbstractOperator:173 - Reconciliation #4868(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:11:05 INFO  AbstractOperator:318 - Reconciliation #4868(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:13:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:13:01 INFO  AbstractOperator:173 - Reconciliation #4869(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:13:06 INFO  AbstractOperator:318 - Reconciliation #4869(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:15:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:15:01 INFO  AbstractOperator:173 - Reconciliation #4870(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:15:05 INFO  AbstractOperator:318 - Reconciliation #4870(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:17:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:17:01 INFO  AbstractOperator:173 - Reconciliation #4871(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:17:05 INFO  AbstractOperator:318 - Reconciliation #4871(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:19:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:19:01 INFO  AbstractOperator:173 - Reconciliation #4872(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:19:04 INFO  AbstractOperator:318 - Reconciliation #4872(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:21:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:21:01 INFO  AbstractOperator:173 - Reconciliation #4873(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:21:06 INFO  AbstractOperator:318 - Reconciliation #4873(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:23:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:23:01 INFO  AbstractOperator:173 - Reconciliation #4874(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:23:04 INFO  AbstractOperator:318 - Reconciliation #4874(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:25:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:25:01 INFO  AbstractOperator:173 - Reconciliation #4875(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:25:04 INFO  AbstractOperator:318 - Reconciliation #4875(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:27:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:27:01 INFO  AbstractOperator:173 - Reconciliation #4876(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:27:05 INFO  AbstractOperator:318 - Reconciliation #4876(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:29:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:29:01 INFO  AbstractOperator:173 - Reconciliation #4877(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:29:05 INFO  AbstractOperator:318 - Reconciliation #4877(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:31:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:31:01 INFO  AbstractOperator:173 - Reconciliation #4878(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:31:06 INFO  AbstractOperator:318 - Reconciliation #4878(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:33:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:33:01 INFO  AbstractOperator:173 - Reconciliation #4879(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:33:05 INFO  AbstractOperator:318 - Reconciliation #4879(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:35:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:35:01 INFO  AbstractOperator:173 - Reconciliation #4880(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:35:06 INFO  AbstractOperator:318 - Reconciliation #4880(timer) Kafka(default/tom-kafka): reconciled
2021-02-26 07:37:01 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace default...
2021-02-26 07:37:01 INFO  AbstractOperator:173 - Reconciliation #4881(timer) Kafka(default/tom-kafka): Kafka tom-kafka should be created or updated
2021-02-26 07:37:05 INFO  AbstractOperator:318 - Reconciliation #4881(timer) Kafka(default/tom-kafka): reconciled
fzyzcjy commented 3 years ago

@scholzj

As for resource limits: No changes to the install.yml. I install by running that install yaml file. As for prometheus: I have netdata, which can be seen in the first post in this thread.

But since it disappears automatically (automagically), I can just ignore this error... And focus on only the memory problem.

sushiMix commented 3 years ago

is there some news on memory problem investigation ?

Excpt0r commented 3 years ago

@scholzj

I can confirm that this bug still exists in strimzi v0.24.0 As you suggested I tried adjusting Xmx, but this seems not to be the problem here. When I set Xmx much lower than the container memory limits, the container memory size still grows, without getting an out of memory Exception.

When I set the pod resource limit to 800Mi, the operator pod is not OOM killed during night, and the memory consumption is settling around 600MB (still slowly growing though). I would definitely suggest to increase the default values (deployment and helm), is there any experience with which configuration to run strimzi in production?

I was not able to debug the container via Jconsole, maybe that's a problem of the base image (headless jre?). So I deployed strimzi operator and did a VM dump (kill -3 on java process) every 60s, along with a ps axu in container and a screenshot of the pod memory curve in prometheus.

What strikes me is the number of threads, and that it is increasing over time, somehow like the memory. That could explain why heap settings are not helping, maybe there is a thread leak that consumes stack memory. I also included the extracted thread names of the first and last VM dump to see a diff, the new threads are named "Executor for Watch".

Therefore I also tested running strimzi with only a single namespace (maybe to reduce watching activities), but that did not help.

strimzi_memory_metrics

strimzi_dumps.tar.gz

mologie commented 3 years ago

We experience the same issue in our K8s 1.21 cluster, where the Strimzi cluster operator version 0.25 is occasionally OOM-killed.

Quick analysis:

The operator is deployed with these limits by default:

    resources:
      limits:
        cpu: "1"
        memory: 384Mi
      requests:
        cpu: 200m
        memory: 384Mi

However I do not see any args being passed to java which would have the JVM respect those limits:

[strimzi@strimzi-cluster-operator-686dddcfb6-sg6f9 strimzi]$ cat /proc/$(pgrep java)/cmdline | tr '\0' '\n' | grep Xm
[strimzi@strimzi-cluster-operator-686dddcfb6-sg6f9 strimzi]$

So it of course will eventually exceed them and get OOM killed.

Note that even if we were to add limits to the java process args they'd have to be a bit lower than the limits set in K8s. This wonderful article sums it up nicely:

https://srvaroa.github.io/jvm/kubernetes/memory/docker/oomkiller/2019/05/29/k8s-and-java.html

tl;dr: Set -Xms384m -Xmx384m, and raise requests and limits to 500Mi.

scholzj commented 3 years ago

AFAIK for 500Mi memory request it should need to be a lot lees than -Xmx384 ... more like 100 or 100 something would be my guess. With the default 384Mi, using something like -Xms80m -Xmx80m made it way more stable for me.

mologie commented 3 years ago

Thanks for the tip @scholzj, I will give that a shot.

Just for reference for those interested, I am patching our Strimzi deployment automatically like so via yq -- you may want to use kustomize instead:

# using yq v4 from https://github.com/mikefarah/yq
curl 'https://strimzi.io/install/latest?namespace=kafka' | \
  yq e 'select(.kind=="Deployment" and .metadata.name=="strimzi-cluster-operator")
    .spec.template.spec.containers[0] |= (
      .env += [
        {"name": "KUBERNETES_SERVICE_DNS_DOMAIN", "value": "my.fancy.cluster.local"},
        {"name": "JAVA_OPTS", "value": "-Xms128m -Xmx128m"}
      ] |
      .resources.requests.memory = "512Mi" |
      .resources.limits.memory = "512Mi"
    )' \
  - > workloads/strimzi.yaml
Excpt0r commented 3 years ago

@scholzj Why not changing the defaults, that it works out of the box and is production ready? Maybe adding the JVM settings to values.yaml of the helm chart would be good, just in case " -Xms80m -Xmx80m" is not working for everyone.

scholzj commented 3 years ago

TBH, I think it still needs some more investigation ... but those are the settings which helped me in my long running cluster.

Excpt0r commented 3 years ago

My solution was to set the limits to 800Mi, works stable so far. We just need to ensure that there is enough non-heap memory left for the JVM stack (threads). If you don't want to define an exakt heapsize, maybe setting -XX:MaxRAMPercentage=10 (default is 25) would help to leave more memory for the JVM stack, but I didn't test that.

But at the end, making the JVM options configurable in values.yaml and do a combination of increase resource limits and set JVM option with less heap (or MaxRAMPercentage) as default values would be a good idea in my opinion.

scholzj commented 3 years ago

Well, it is configurable. You just need to add the JAVA_OPTS environment variable tot he container. For example using the command from @mologie

svonliebenstein commented 2 years ago

We're running into the same problem as well and because we use the helm chart, we can't use the workaround as suggested by @mologie. If I'm correct, the helm chart doesn't allow for passing any JAVA_OPTS to the operator, this makes it quite difficult to prevent the OOM exceptions.

For example when using nodes with 16GB memory available, the java process will think it can use 25% of this for the heap size. As the current memory limit is hardcoded with 384Mi, eventually the operator will reach this limit and will be killed.

scholzj commented 2 years ago

@svonliebenstein I don't understand what part does the Helm chart not support. If setting resource requests or JAVA_OPTS env var. For the env var, #5859 might help i the next release.

svonliebenstein commented 2 years ago

I didn't see #5859, that would indeed make it possible to set the correct JAVA_OPTS. Thanks for the suggestion :)

scholzj commented 2 years ago

@svonliebenstein It is not released yet. But I hope that the 0.27.0 release might happen some time during December.

cooervo commented 8 months ago

If anyone encounters this issue from 2021 there's a newer update, Dec 2023, from scholzj regarding kafka memory consumption: https://github.com/orgs/strimzi/discussions/9445#discussioncomment-7794983