search

orientechnologies / orientdb

OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text and Geospatial models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries.
https://orientdb.dev
Apache License 2.0
4.75k stars 871 forks source link

Error on reconnecting to distributed database #9394

Open knuthelgesen opened 4 years ago

knuthelgesen commented 4 years ago

OrientDB Version: 3.1.2, running in Docker based on official Docker image

Java Version: 1.8.0_265

OS: Docker host is SLES-12

Expected behavior

The clients can reconnect to a distributed database with a connection string specifying all the hostsnames.

Actual behavior

We have three Docker hosts, and on each one there is a container running OrientDB. These are all part of the same cluster, which works fine. The client connects to the servers using the code: new OrientGraphFactory("remote:docker01;docker02;docker03/portal "user", "pw").setupPool(1, 50) This also works fine.

However, after a while of inactivity, the connection from the client to the server goes stale or times out. And the client is not able to reconnect to the server.

We see the following error message in the client log: 2020-09-16 18:18:43.169 INFO 16 --- [nio-8080-exec-2] c.o.orient.client.remote.OStorageRemote : Caught Network I/O errors on docker03:2424/portal, trying an automatic reconnection... (error: null) 2020-09-16 18:18:43.209 ERROR 16 --- [nio-8080-exec-2] n.k.b.p.rest.controller.BildeController : Feilmelding: null com.orientechnologies.common.io.OIOException: null at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:497) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:365) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:405) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1095) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:72) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:76) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:84) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:34) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.query(ODatabaseDocumentAbstract.java:358) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:769) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.tinkerpop.blueprints.impls.orient.OrientGraphQuery.vertices(OrientGraphQuery.java:206) ~[orientdb-graphdb-3.1.2.jar!/:3.1.2] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertices(OrientBaseGraph.java:860) ~[orientdb-graphdb-3.1.2.jar!/:3.1.2] at no.kommune.bergen.plattform.transaksjon.TransactionWrapper.findByProperty(TransactionWrapper.java:130) ~[domeneplattform-12.0-SNAPSHOT.jar!/:na] at no.kommune.bergen.plattform.transaksjon.TransactionWrapper.findById(TransactionWrapper.java:138) ~[domeneplattform-12.0-SNAPSHOT.jar!/:na] at no.kommune.bergen.portal.rest.controller.BildeController.get(BildeController.java:53) ~[classes!/:na] at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at no.kommune.bergen.portal.filter.SiteFilter.doFilter(SiteFilter.java:47) [portal-domene-12.0-SNAPSHOT.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at no.kommune.bergen.portal.filter.TransactionFilter.doFilter(TransactionFilter.java:33) [portal-domene-12.0-SNAPSHOT.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:836) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1747) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: java.io.EOFException: null at java.io.DataInputStream.readByte(DataInputStream.java:267) ~[na:1.8.0_212] at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.readByte(OChannelBinary.java:78) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:185) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:169) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2221) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:385) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:447) ~[orientdb-client-3.1.2.jar!/:3.1.2] ... 73 common frames omitted

And the following error in the server log: Error on unmarshalling content. Class: q com.orientechnologies.orient.core.exception.ODatabaseException: The database instance is not set in the current thread. Be sure to set it with: ODatabaseRecordThreadLocal.instance().set(db); at com.orientechnologies.orient.core.db.ODatabaseRecordThreadLocal.get(ODatabaseRecordThreadLocal.java:61) at com.orientechnologies.orient.core.sql.query.OSQLQuery.deserializeQueryParameters(OSQLQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.queryFromStream(OSQLQuery.java:137) at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.queryFromStream(OSQLSynchQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.fromStream(OSQLQuery.java:109) at com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:86) at com.orientechnologies.orient.client.remote.message.OCommandRequest.read(OCommandRequest.java:70) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:319) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:67) Error reading request com.orientechnologies.orient.core.exception.OSerializationException: Error on unmarshalling content. Class: q at com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:92) at com.orientechnologies.orient.client.remote.message.OCommandRequest.read(OCommandRequest.java:70) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:319) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:67) Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: The database instance is not set in the current thread. Be sure to set it with: ODatabaseRecordThreadLocal.instance().set(db); at com.orientechnologies.orient.core.db.ODatabaseRecordThreadLocal.get(ODatabaseRecordThreadLocal.java:61) at com.orientechnologies.orient.core.sql.query.OSQLQuery.deserializeQueryParameters(OSQLQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.queryFromStream(OSQLQuery.java:137) at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.queryFromStream(OSQLSynchQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.fromStream(OSQLQuery.java:109) at com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:86) ... 4 more

Steps to reproduce

  1. Set up a distributed database with three nodes.
  2. Connect using all the hostsnames in the connectionstring.
  3. Wait for the connection to time out.
  4. Attempt reconnection.
lvca commented 3 years ago

This issue should be fixed in the most recent version. Please use the latest 3.1.x.