search

eclipse-vertx / vert.x

Vert.x is a tool-kit for building reactive applications on the JVM
http://vertx.io
Other
14.32k stars 2.08k forks source link

Thread blocked for long time #3021

Open RaiBnod opened 5 years ago

RaiBnod commented 5 years ago

What is the issue is related too? And what is the possible way to solve it?

[hz.nubeio-cluster.priority-generic-operation.thread-0] WARN  c.h.s.i.o.impl.Invocation - [172.22.0.7]:5701 [dev] [3.10.6] Retrying invocation: Invocation{op=com.hazelcast.internal.partition.operation.PromotionCommitOperation{serviceName='hz:core:partitionService', identityHash=2086216157, partitionId=-1, replicaIndex=0, callId=1106647, invocationTime=1562121161640 (2019-07-03 02:32:41.640), waitTimeout=-1, callTimeout=5000}, tryCount=2147483647, tryPauseMillis=500, invokeCount=67910, callTimeoutMillis=5000, firstInvocationTimeMs=1562096649943, firstInvocationTime='2019-07-02 19:44:09.943', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[172.22.0.7]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.spi.exception.RetryableHazelcastException: Another promotion is being run currently. This is only expected when promotion is retried to an unresponsive destination.
2019-07-03 02:32:42 [vertx-blocked-thread-checker] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-worker-thread-8,5,main] has been blocked for 24072923 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
    at java.lang.Thread.sleep(Native Method)
    at com.hazelcast.internal.partition.impl.InternalPartitionServiceImpl.getPartitionOwnerOrWait(InternalPartitionServiceImpl.java:235)
    at com.hazelcast.internal.partition.impl.InternalPartitionServiceImpl.getMemberPartitionsMap(InternalPartitionServiceImpl.java:989)
    at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnAllPartitions(OperationServiceImpl.java:375)
    at com.hazelcast.multimap.impl.MultiMapProxySupport.entrySetInternal(MultiMapProxySupport.java:164)
    at com.hazelcast.multimap.impl.ObjectMultiMapProxy.entrySet(ObjectMultiMapProxy.java:182)
    at io.vertx.spi.cluster.hazelcast.impl.HazelcastAsyncMultiMap.lambda$removeAllMatching$0(HazelcastAsyncMultiMap.java:76)
    at io.vertx.spi.cluster.hazelcast.impl.HazelcastAsyncMultiMap$$Lambda$251/1786991293.handle(Unknown Source)
    at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:272)
    at io.vertx.core.impl.ContextImpl$$Lambda$48/1783083399.run(Unknown Source)
    at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
    at io.vertx.core.impl.TaskQueue$$Lambda$45/1368173251.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
tsegismont commented 5 years ago

Which versions of the Hazelcast cluster manager and Hazelcast library do you use? We've fixed a lot of issues with blocking calls on the event loop in recent versions.

RaiBnod commented 5 years ago

Which versions of the Hazelcast cluster manager and Hazelcast library do you use? We've fixed a lot of issues with blocking calls on the event loop in recent versions.

verx-hazelcast-3.6.0 is used.

tsegismont commented 5 years ago

In 3.6 the fixes were present. I took another look at the log and stacktrace. Indeed the warning is related to a worker thread.

I would recommend to put together more logs from Hazelcast and ask their team on Gitter what can could be going on with the nodes.

Le mer. 3 juil. 2019 à 11:49, Binod notifications@github.com a écrit :

Which versions of the Hazelcast cluster manager and Hazelcast library do you use? We've fixed a lot of issues with blocking calls on the event loop in recent versions.

verx-hazelcast-3.6.0 is used.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/eclipse-vertx/vert.x/issues/3021?email_source=notifications&email_token=AALOLNUPGZGIY7ZODDKYG4LP5RYZDA5CNFSM4H5BS34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZD5KIY#issuecomment-508024099, or mute the thread https://github.com/notifications/unsubscribe-auth/AALOLNTDFVSIZA4M23I3GY3P5RYZDANCNFSM4H5BS34A .