search

reactiverse / consul-cluster-manager

Consul - based cluster manager that can be plugged into Vert.x ecosystem.
Apache License 2.0
20 stars 3 forks source link

issue with "No handlers for address" #23

Open deedarb opened 3 years ago

deedarb commented 3 years ago

i have strange error with clustered event bus handler (using reactiverse/consul-cluster-manager and vertx 3.9.5)

io.vertx.core.eventbus.ReplyException: No handlers for address NotifierService
    at io.vertx.core.eventbus.impl.EventBusImpl.deliverMessageLocally(EventBusImpl.java:411)
    at io.vertx.core.eventbus.impl.EventBusImpl.deliverMessageLocally(EventBusImpl.java:360)
    at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.sendToSubs(ClusteredEventBus.java:347)
    at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.onSubsReceived(ClusteredEventBus.java:235)
    at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.lambda$sendOrPub$8(ClusteredEventBus.java:227)
    at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:105)
    at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:150)
    at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:111)
    at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:176)
    at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:21)
    at io.vertx.core.impl.SucceededFuture.onComplete(SucceededFuture.java:41)
    at io.vertx.core.Future.lambda$compose$3(Future.java:373)
    at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:105)
    at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:150)
    at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:111)
    at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:176)
    at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:21)
    at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:105)
    at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:150)
    at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:111)
    at io.vertx.spi.cluster.consul.impl.ConsulAsyncMultiMap.lambda$doGet$16(ConsulAsyncMultiMap.java:201)
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327)
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) 
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
    at java.lang.Thread.run(Thread.java:834) [?:?]

where should i dig to find cause? asked for help in https://gitter.im/eclipse-vertx/vertx-users - no response :| may be app sometimes leaves cluster and that causes exception, how can i prove/fix that? is it somehow network related? consul and apps on same machine

deedarb commented 3 years ago

error occurrence is not regular, after investigating logs: exceptions in february month found in logs from 4th until 11th, this is strange... i have eight services running on two VM's and one consul agent running on one of VM.

deedarb commented 3 years ago

https://groups.google.com/g/vertx/c/3IVhc5DJNlA

there is answer from Thomas (tsegismont)

OK so it happens when you restart nodes, and I guess you restart a lot of them at the same time. One way to avoid loosing clustered data (event bus subscriptions and haInfo) is to increase the number of backups for the corresponding multimap (vertx.subs) and map (vertx.haInfo). Beware that versions up to 3.3.3 have a bug in the default cluster setup file (https://github.com/vert-x3/vertx-hazelcast/commit/4b6e6f815a221306ae93c500b5e62d8c1452397a)

We are working for a fix in 3.4

it could be cause for problem because during updating apps we have bash script which stops all apps and copies new jars and starts them, maybe old subs are left in consul __vertx.subs/haInfo