PersistenceSubscription - Idle loosing connection

[jselamy]

I came to realized that after period of inactivity on a persitentSubscription. The consumer looses the connection, Is there a way to prevent such a thing ?

Any ideas ?

Regards.

[msemys]

by default driver tries to reconnect 10 times. try to increase or set unlimited (-1) e.g.: EventStoreBuilder.newBuilder().maxReconnections(-1)

[jselamy]

Sweet! i'll try this.

Regards.

[jselamy]

Hi,

I need some help here, because i've did what you suggested but with no luck. Once the connection goes idle the persistence subscription is lost

[jselamy]

Any news on that subject ?

[msemys]

I haven't got a chance to look at it.

• what esjc version you are using?
• what es version you are using?
• what is SubscriptionDropReason or Exception in PersistentSubscriptionListener.onClose(...)?
• what esjc logs shows?

[jselamy]

Thanks for the reply

• what esjc version you are using? 2.0.0
• what es version you are using? 5.0.0.0
• what is SubscriptionDropReason or Exception in PersistentSubscriptionListener.onClose(...)? Ill need to log it and see
• what esjc logs shows? currently not logging out anything. i'll activate the logs

[msemys]

also try with the latest esjc version 2.1.0 - there are some fixes related with client reconnect

[jselamy]

This is the stack trace i get.

2019-04-26 14:02:44.282 ERROR 8 --- [         es-1-6] c.e.i.e.c.s.p.PersistentSubscriber       : Persistent subscription closed
10:02:44.283: [APP/PROC/WEB.0] com.github.msemys.esjc.ConnectionClosedException: Connection was closed.
10:02:44.283: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.AbstractSubscriptionOperation.connectionClosed(AbstractSubscriptionOperation.java:220) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.283: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.lambda$purgeSubscribedAndDropped$2(SubscriptionManager.java:57) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.283: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.purgeSubscribedAndDropped(SubscriptionManager.java:56) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.onTcpConnectionClosed(EventStoreTcp.java:812) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.lambda$null$12(EventStoreTcp.java:865) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1152) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:768) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:744) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1350) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.tcp.handler.HeartbeatHandler.lambda$userEventTriggered$0(HeartbeatHandler.java:61) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0] com.github.msemys.esjc.ConnectionClosedException: Connection was closed.
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.AbstractSubscriptionOperation.connectionClosed(AbstractSubscriptionOperation.java:220) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.lambda$purgeSubscribedAndDropped$2(SubscriptionManager.java:57) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.purgeSubscribedAndDropped(SubscriptionManager.java:56) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.onTcpConnectionClosed(EventStoreTcp.java:812) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.lambda$null$12(EventStoreTcp.java:865) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1152) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:768) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:744) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1350) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.tcp.handler.HeartbeatHandler.lambda$userEventTriggered$0(HeartbeatHandler.java:61) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0] 2019-04-26 14:02:44.282 ERROR 8 --- [         es-1-7] c.e.i.e.c.s.p.PersistentSubscriber       : Persistent subscription closed
10:02:44.284: [APP/PROC/WEB.0] com.github.msemys.esjc.ConnectionClosedException: Connection was closed.
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.AbstractSubscriptionOperation.connectionClosed(AbstractSubscriptionOperation.java:220) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.lambda$purgeSubscribedAndDropped$2(SubscriptionManager.java:57) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.purgeSubscribedAndDropped(SubscriptionManager.java:56) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.onTcpConnectionClosed(EventStoreTcp.java:812) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.lambda$null$12(EventStoreTcp.java:865) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1152) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:768) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:744) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1350) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.tcp.handler.HeartbeatHandler.lambda$userEventTriggered$0(HeartbeatHandler.java:61) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.284: [APP/PROC/WEB.0]  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0] 2019-04-26 14:02:44.282 ERROR 8 --- [         es-1-2] c.e.i.e.c.s.p.PersistentSubscriber       : Persistent subscription closed
10:02:44.284: [APP/PROC/WEB.0] com.github.msemys.esjc.ConnectionClosedException: Connection was closed.
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.AbstractSubscriptionOperation.connectionClosed(AbstractSubscriptionOperation.java:220) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.lambda$purgeSubscribedAndDropped$2(SubscriptionManager.java:57) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_212]
10:02:44.284: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_212]
10:02:44.285: [APP/PROC/WEB.0]  at com.github.msemys.esjc.subscription.manager.SubscriptionManager.purgeSubscribedAndDropped(SubscriptionManager.java:56) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.285: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.onTcpConnectionClosed(EventStoreTcp.java:812) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.285: [APP/PROC/WEB.0]  at com.github.msemys.esjc.EventStoreTcp.lambda$null$12(EventStoreTcp.java:865) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1152) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:768) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:744) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1350) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at com.github.msemys.esjc.tcp.handler.HeartbeatHandler.lambda$userEventTriggered$0(HeartbeatHandler.java:61) ~[esjc-2.1.0.jar!/:2.1.0]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
10:02:44.285: [APP/PROC/WEB.0]  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]

[msemys]

connection was closed due to heartbeat timeout. By default it is 1500 milliseconds, try to increase it (also you could try to send less heartbeat request, by increasing its interval) e.g.:

EventStoreBuilder.newBuilder()
  .heartbeatTimeout(Duration.ofSeconds(3))
  .heartbeatInterval(Duration.ofSeconds(2))