search

neo4j / neo4j-java-driver

Neo4j Bolt driver for Java
Apache License 2.0
329 stars 155 forks source link

[Bug] Exception Handling when switching leader nodes using neo4j scheme #755

Closed sachinsudhi closed 3 years ago

sachinsudhi commented 4 years ago

Full Exception stack trace:

WARNING [LoadBalancer] (Neo4jDriverIO-3-9) Failed to obtain a connection towards address x.x.x.x:7687: org.neo4j.driver.exceptions.SessionExpiredException: Server at x.x.x.x:7687 is no longer available
at org.neo4j.driver.internal.cluster.loadbalancing.LoadBalancer.lambda$acquire$9(LoadBalancer.java:203)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.neo4j.driver.internal.util.Futures.lambda$asCompletionStage$0(Futures.java:97)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:501)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.neo4j.driver.exceptions.ServiceUnavailableException: Unable to connect to x.x.x.x:7687, ensure the database is running and that there is a working network connection to it.
at org.neo4j.driver.internal.async.connection.ChannelConnectedListener.databaseUnavailableError(ChannelConnectedListener.java:76)
at org.neo4j.driver.internal.async.connection.ChannelConnectedListener.operationComplete(ChannelConnectedListener.java:70)
at org.neo4j.driver.internal.async.connection.ChannelConnectedListener.operationComplete(ChannelConnectedListener.java:37)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
... 4 more
Caused by: org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /x.x.x.x:7687
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.neo4j.driver.internal.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

2020-08-13 12:27:21,927 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /customer/stream/account/104130202 failed, error id: a7be2253-6f97-46ae-b029-811494561310-1: org.jboss.resteasy.spi.UnhandledException: org.neo4j.driver.exceptions.SessionExpiredException: Failed to obtain connection towards WRITE server. Known routing table is: Ttl 1597336273867, currentTime 1597336041912, routers AddressSet=[y.y.y.y:7687, z.z.z.z:7687], writers AddressSet=[], readers AddressSet=[z.z.z.z:7687, y.y.y.y:7687], database ''
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:216)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:515)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:259)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:160)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:163)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:245)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:132)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:37)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:94)
at io.quarkus.runtime.CleanableExecutor$CleaningRunnable.run(CleanableExecutor.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: org.neo4j.driver.exceptions.SessionExpiredException: Failed to obtain connection towards WRITE server. Known routing table is: Ttl 1597336273867, currentTime 1597336041912, routers AddressSet=[y.y.y.y:7687, z.z.z.z:7687], writers AddressSet=[], readers AddressSet=[z.z.z.z:7687, y.y.y.y:7687], database ''
at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:98)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:92)
at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.newOrExistingNativeTransaction(BoltTransaction.java:62)
at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.(BoltTransaction.java:50)
at org.neo4j.ogm.drivers.bolt.driver.BoltDriver.lambda$null$0(BoltDriver.java:128)
at org.neo4j.ogm.session.transaction.DefaultTransactionManager.openTransaction(DefaultTransactionManager.java:75)
at org.neo4j.ogm.session.Neo4jSession.beginTransaction(Neo4jSession.java:536)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:586)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:564)
at org.neo4j.ogm.session.delegates.ExecuteQueriesDelegate.executeAndMap(ExecuteQueriesDelegate.java:138)
at org.neo4j.ogm.session.delegates.ExecuteQueriesDelegate.query(ExecuteQueriesDelegate.java:111)
at org.neo4j.ogm.session.Neo4jSession.query(Neo4jSession.java:425)
at com.service.AccountServiceImpl.transformAndReturnAccountNode(AccountServiceImpl.java:32)
at com.service.AccountServiceImpl_ClientProxy.transformAndReturnAccountNode(AccountServiceImpl_ClientProxy.zig:156)
at com.controller.StreamController.getAccountDetails(StreamController.java:33)
at com.controller.StreamController_Subclass.getAccountDetails$$superaccessor1(StreamController_Subclass.zig:273)
at com.controller.StreamController_Subclass$$function$$1.apply(StreamController_Subclass$$function$$1.zig:33)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:54)
at io.smallrye.metrics.interceptors.CountedInterceptor.countedCallable(CountedInterceptor.java:95)
at io.smallrye.metrics.interceptors.CountedInterceptor.countedMethod(CountedInterceptor.java:70)
at io.smallrye.metrics.interceptors.CountedInterceptor_Bean.intercept(CountedInterceptor_Bean.zig:366)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:50)
at io.smallrye.metrics.interceptors.TimedInterceptor.timedCallable(TimedInterceptor.java:95)
at io.smallrye.metrics.interceptors.TimedInterceptor.timedMethod(TimedInterceptor.java:70)
at io.smallrye.metrics.interceptors.TimedInterceptor_Bean.intercept(TimedInterceptor_Bean.zig:366)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:41)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:32)
at com.controller.StreamController_Subclass.getAccountDetails(StreamController_Subclass.zig:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:638)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:504)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:454)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:456)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:417)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:391)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:488)
... 20 more
Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause
at org.neo4j.driver.internal.cluster.loadbalancing.LoadBalancer.acquire(LoadBalancer.java:190)
at org.neo4j.driver.internal.cluster.loadbalancing.LoadBalancer.lambda$null$8(LoadBalancer.java:206)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
michael-simons commented 4 years ago

+EDIT+

Hi @sachinsudhi

I mixed up the repos, I first responded to your issue in OGM, now I realize we're talking about the driver itself.

We have two options here: When you use idempotent, transactional functions as described in the manual: https://neo4j.com/docs/driver-manual/current/session-api/simple/#driver-simple-transaction-fn our internal transaction manager does exactly what you're requesting. As I guess you're working in Quarkus environment, I really recommend using the pure Java driver with transactional functions and do the mapping into domain objects explicitly ("by hand"). If the domain model is small enough, you would greatly benefit.

In case of Neo4j-OGM we cannot use them, as it abstract over a couple of different transports and participates in application transactions if available. In such cases we recommend using Resillience4j as @meistermeier demonstrated here: https://github.com/meistermeier/resilience4j-neo4j-ogm/blob/master/src/test/java/org/meistermeier/neo4j/resilience4j/ogm/SessionWithRetryTest.java

Would that help in your case?

gjmwoods commented 3 years ago

Hi @sachinsudhi. It's been a while since we've heard anything back so we'll close the issue for now (sorry I'm not sure if you responded to the issue on the other repo that @michael-simons mentioned). Please feel free to reopen if we can provide any more help.