search

vert-x3 / issues

Apache License 2.0
37 stars 7 forks source link

Running with ignite causes vertx application to hang #434

Open ramzee2017 opened 5 years ago

ramzee2017 commented 5 years ago

The description of the problem: My setup looks like the following. I have a bunch of microservices implemented using Vertx Java and deployed on docker swarm consisting of 3 manager and 2 worker nodes. Apache Ignite is used as the cluster manager solution for the different microservices, potentially running on different swarm nodes, to be able to send messages to each other over the event bus. Once in while, after working with my microservices, I start getting the exceptions below from all of my microservices. My application is not accessible during that time but after a few minutes of inactivity it comes back till next time I start getting the same behaviour.

 Dec 16, 2018 9:22:22 AM io.vertx.core.impl.BlockedThreadChecker

     WARNING: Thread Thread[vert.x-worker-thread-8,5,main] has been blocked for 545826 ms, time limit is 60000
   io.vertx.core.VertxException: Thread blocked

    at sun.misc.Unsafe.park(Native Method)

    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)

    at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)

    at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)

    at org.apache.ignite.internal.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)

    at io.vertx.spi.cluster.ignite.impl.AsyncMultiMapImpl.lambda$execute$7(AsyncMultiMapImpl.java:228)

            at io.vertx.spi.cluster.ignite.impl.AsyncMultiMapImpl$$Lambda$261/695865484.handle(Unknown Source)

    at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:273)

    at io.vertx.core.impl.ContextImpl$$Lambda$25/1823409783.run(Unknown Source)

    at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)

    at io.vertx.core.impl.TaskQueue$$Lambda$24/1758008124.run(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.lang.Thread.run(Thread.java:748)

It looks like all of the microservices are stuck doing some ignite related activity. I wonder if this is a known issue and what are the possible ways to solve it e.g.: ignite or vertx specific configuration that might help here, etc.

Thanks, Ilya.

bysensa commented 5 years ago

@ramzee2017 Hi. As far as I understand your problem is that the Apache Ignite instance starts after vertx creates a BlockedThreadChecker instance that defines a thread lock at the time when the IgniteClusterManager instance is called by the join method in which the Ignite instance is created. Ignite takes a few seconds to start, depending on the configuration. You can either create an instance Ignite before the start of the vertx and pass it to an instance in IgniteClusterManager or change the delay time of the response to a long thread blocking.