vietj
commented
5 months ago I see you reported another blocked error (DNS) based on Kotlin, I am wondering if Kotlin coroutines is not related to that instead of Vert.x
Open sishbi opened 5 months ago
vietj
commented
5 months ago I see you reported another blocked error (DNS) based on Kotlin, I am wondering if Kotlin coroutines is not related to that instead of Vert.x
sishbi
commented
5 months ago I see you reported another blocked error (DNS) based on Kotlin, I am wondering if Kotlin coroutines is not related to that instead of Vert.x
For the SQL client issue https://github.com/eclipse-vertx/vertx-sql-client/issues/1429, the blocked thread is during a class-load triggered by a DNS lookup in Netty code
For this issue the blocked thread originated in io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll invoking something on an executor
I don't know if Kotlin coroutines is making this worse or not - but as this is within the Vertx/Netty code there isn't much we can do from our code...
sishbi
commented
5 months ago Some more stacks: This is from 2 pods starting at the same time, so some of the stacks may be repeated...
pod1:
Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2388 ms, time limit is 2000 ms
Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3387 ms, time limit is 2000 ms
pod2:
Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2817 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:1553)
at java.base/java.lang.System$2.start(System.java:2577)
at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.submitTaskWhenStarted(KafkaReadStreamImpl.java:123)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.start(KafkaReadStreamImpl.java:116)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.subscribe(KafkaReadStreamImpl.java:443)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:197)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:186)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:180)
at com.kodypay.lychee.kafka.KafkaConsumer.<init>(KafkaConsumer.kt:57)
at com.kodypay.cards.kafka.payments.SettlementConsumer.<init>(SettlementConsumer.kt:34)
at com.kodypay.cards.kafka.KafkaVerticle.start(KafkaVerticle.kt:37)
at io.vertx.kotlin.coroutines.CoroutineVerticle$start$1.invokeSuspend(CoroutineVerticle.kt:53)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:363)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)io.vertx.core.VertxException: Thread blocked
at java.base/jdk.internal.misc.Unsafe.unpark(Native Method)
at java.base/java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:181)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.signalNext(AbstractQueuedSynchronizer.java:645)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1060)
at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
at java.base/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:179)
at java.base/java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:423)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll(KafkaReadStreamImpl.java:876)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:683)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:695)
at com.kodypay.lychee.kafka.KafkaConsumer$3.invokeSuspend(KafkaConsumer.kt:65)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:65)
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:371)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)io.vertx.core.VertxException: Thread blocked
at java.base/jdk.internal.misc.Unsafe.unpark(Native Method)
at java.base/java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:181)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.signalNext(AbstractQueuedSynchronizer.java:645)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1060)
at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
at java.base/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:179)
at java.base/java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:423)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll(KafkaReadStreamImpl.java:876)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:683)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:695)
at com.kodypay.lychee.kafka.KafkaConsumer$3.invokeSuspend(KafkaConsumer.kt:65)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:65)
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:371)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)And another:
Note: this error occurred about 3 minutes after the k8s pod started. So it is likely the first time the consumer joined the kafka cluster.
Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2018 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
at java.base/java.security.AccessController.getStackAccessControlContext(Native Method)
at java.base/java.security.AccessController.getContext(AccessController.java:1006)
at java.base/java.lang.Thread.<init>(Thread.java:741)
at java.base/java.lang.Thread.<init>(Thread.java:1275)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.lambda$start$0(KafkaReadStreamImpl.java:115)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.submitTaskWhenStarted(KafkaReadStreamImpl.java:123)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.start(KafkaReadStreamImpl.java:116)
at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.subscribe(KafkaReadStreamImpl.java:443)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:197)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:186)
at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:180)
at com.kodypay.lychee.kafka.KafkaConsumer.<init>(KafkaConsumer.kt:57)
at com.kodypay.cards.kafka.payments.CombinedPaymentEventConsumer.<init>(CombinedPaymentEventConsumer.kt:26)
at com.kodypay.cards.kafka.KafkaVerticle.start(KafkaVerticle.kt:43)
at io.vertx.kotlin.coroutines.CoroutineVerticle$start$1.invokeSuspend(CoroutineVerticle.kt:53)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:363)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Version
4.5.4
Context
Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2748 ms, time limit is 2000 msDo you have a reproducer?
No
Steps to reproduce
Extra
OS: Amazon Linux (Docker) running in EKS (k8s) JDK: 21, eclipse-temurin:21-jdk