syndesis-db out of available connections after ~week of uptime

maschmid commented 6 years ago

This is a...


[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[X] Bug report  
[ ] Documentation issue or request

The problem

After about a week of uptime, with ~2M messages, the syndesis-db starts to log the following errors

FATAL: remaining connection slots are reserved for non-replication superuser connections

and any request to the backend fails, e.g.:

2018-05-14 07:20:49.847 ERROR [-,0bae21f38eea0c62,0bae21f38eea0c62,false] 1 --- [  XNIO-3 task-5] .s.e.v.h.e.SyndesisServerExceptionMapper : Internal Server Exception. org.postgresql.util.PSQLException: The connection attempt failed.

org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: org.postgresql.util.PSQLException: The connection attempt failed.
    at org.skife.jdbi.v2.DBI.open(DBI.java:229) ~[jdbi-2.78.jar!/:2.78]
    at io.syndesis.server.jsondb.impl.SqlJsonDB.getAsStreamingOutput(SqlJsonDB.java:184) ~[server-jsondb-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.jsondb.JsonDB.getAsStream(JsonDB.java:160) ~[server-jsondb-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.jsondb.JsonDB.getAsByteArray(JsonDB.java:130) ~[server-jsondb-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.jsondb.dao.JsonDbDao.fetchAll(JsonDbDao.java:96) ~[server-jsondb-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.dao.manager.DataManager.lambda$fetchAll$0(DataManager.java:211) ~[server-dao-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.dao.manager.DataManager.doWithDataAccessObject(DataManager.java:430) ~[server-dao-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.dao.manager.DataManager.fetchAll(DataManager.java:211) ~[server-dao-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.endpoint.v1.handler.BaseHandler.fetchAll(BaseHandler.java:46) ~[server-endpoint-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at io.syndesis.server.endpoint.v1.handler.connection.ConnectionHandler.list(ConnectionHandler.java:106) ~[server-endpoint-1.3.5.fuse-000002-redhat-1.jar!/:1.3.5.fuse-000002-redhat-1]
    at sun.reflect.GeneratedMethodAccessor942.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) ~[resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:294) ~[resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:248) ~[resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:235) ~[resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) [resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) [resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.1.4.Final.jar!/:3.1.4.Final]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar!/:3.1.0]
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) [spring-boot-actuator-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:186) [spring-cloud-sleuth-core-1.2.5.RELEASE.jar!/:1.2.5.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:332) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-1.4.21.Final.jar!/:1.4.21.Final]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:215) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.postgresql.Driver.makeConnection(Driver.java:404) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.postgresql.Driver.connect(Driver.java:272) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:310) ~[tomcat-jdbc-8.5.23.jar!/:na]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:203) ~[tomcat-jdbc-8.5.23.jar!/:na]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:735) ~[tomcat-jdbc-8.5.23.jar!/:na]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:667) ~[tomcat-jdbc-8.5.23.jar!/:na]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) ~[tomcat-jdbc-8.5.23.jar!/:na]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:132) ~[tomcat-jdbc-8.5.23.jar!/:na]
    at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:34) ~[jdbi-2.78.jar!/:2.78]
    at org.skife.jdbi.v2.DBI.open(DBI.java:211) ~[jdbi-2.78.jar!/:2.78]
    ... 124 common frames omitted
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_171]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_171]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_171]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_171]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
    at org.postgresql.core.PGStream.<init>(PGStream.java:61) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    ... 136 common frames omitted

Expected behavior

Screenshot

Request and Response Data

API Endpoints and Schemas

Tasks involved / Steps to Reproduce

1. 2. 3. 4.

maschmid commented 6 years ago

select * from pg_stat_activity;

 datid | datname  |  pid   | usesysid | usename  | application_name | client_addr | client_hostname | client_port |         backend_start         |          xact_start           |          query_start          |         state_change          | waiting |        state        | backend_xid | backend_xmin |                                                                                                                                                                                                                     query                                                                                                                                                                                                                      
-------+----------+--------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+---------------------+-------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 16385 | syndesis |  93134 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55320 | 2018-05-12 14:39:48.672298+00 | 2018-05-14 07:19:23.173912+00 | 2018-05-14 07:19:23.173912+00 | 2018-05-14 07:19:23.173947+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |   9049 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54410 | 2018-05-10 10:51:52.138586+00 | 2018-05-14 07:18:13.221704+00 | 2018-05-14 07:18:13.222016+00 | 2018-05-14 07:18:13.222017+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93135 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55324 | 2018-05-12 14:39:53.674097+00 | 2018-05-14 07:07:53.158806+00 | 2018-05-14 07:07:53.158806+00 | 2018-05-14 07:07:53.158811+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |   9051 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54414 | 2018-05-10 10:51:52.151252+00 | 2018-05-10 10:59:41.944064+00 | 2018-05-10 10:59:41.94759+00  | 2018-05-10 10:59:41.949343+00 | f       | idle in transaction |             |         1022 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 16385 | syndesis |  93136 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55332 | 2018-05-12 14:39:55.487785+00 | 2018-05-14 07:20:32.74204+00  | 2018-05-14 07:20:32.74204+00  | 2018-05-14 07:20:32.74222+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |   9053 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54418 | 2018-05-10 10:51:52.162799+00 | 2018-05-10 10:58:40.970412+00 | 2018-05-10 10:58:40.973424+00 | 2018-05-10 10:58:40.975232+00 | f       | idle in transaction |             |          961 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 16385 | syndesis |   9054 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54420 | 2018-05-10 10:51:52.169349+00 | 2018-05-10 10:56:09.285759+00 | 2018-05-10 10:56:09.28897+00  | 2018-05-10 10:56:09.290412+00 | f       | idle in transaction |             |          875 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 16385 | syndesis |   9055 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54422 | 2018-05-10 10:51:52.175493+00 | 2018-05-10 10:54:39.375139+00 | 2018-05-10 10:54:39.378222+00 | 2018-05-10 10:54:39.380498+00 | f       | idle in transaction |             |          851 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 16385 | syndesis |   9056 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54424 | 2018-05-10 10:51:52.181441+00 | 2018-05-10 10:57:15.978365+00 | 2018-05-10 10:57:15.982416+00 | 2018-05-10 10:57:15.988363+00 | f       | idle in transaction |             |          905 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 16385 | syndesis |   9057 |    16384 | syndesis |                  | 10.128.0.1  |                 |       54426 | 2018-05-10 10:51:52.187601+00 | 2018-05-10 10:53:09.62902+00  | 2018-05-10 10:53:09.637176+00 | 2018-05-10 10:53:09.650061+00 | f       | idle in transaction |             |          840 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 16385 | syndesis |  93148 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55348 | 2018-05-12 14:40:03.677819+00 | 2018-05-14 07:22:33.183319+00 | 2018-05-14 07:22:33.183319+00 | 2018-05-14 07:22:33.183502+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93294 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55592 | 2018-05-12 14:41:48.675501+00 | 2018-05-14 07:07:18.152579+00 | 2018-05-14 07:26:43.718983+00 | 2018-05-14 07:26:43.719173+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis | 121029 |    16384 | syndesis |                  | 10.128.0.1  |                 |       46258 | 2018-05-11 13:52:21.361633+00 | 2018-05-14 07:09:18.192629+00 | 2018-05-14 07:09:18.193218+00 | 2018-05-14 07:09:18.193219+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis | 121030 |    16384 | syndesis |                  | 10.128.0.1  |                 |       46264 | 2018-05-11 13:52:23.903255+00 | 2018-05-14 07:07:23.154201+00 | 2018-05-14 07:07:23.154869+00 | 2018-05-14 07:07:23.15487+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93161 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55366 | 2018-05-12 14:40:08.672223+00 | 2018-05-14 07:12:53.164835+00 | 2018-05-14 07:12:53.164835+00 | 2018-05-14 07:12:53.165071+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93162 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55370 | 2018-05-12 14:40:13.670078+00 | 2018-05-14 07:25:38.184107+00 | 2018-05-14 07:25:38.184107+00 | 2018-05-14 07:25:38.184111+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93174 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55390 | 2018-05-12 14:40:18.671928+00 | 2018-05-14 07:14:38.175196+00 | 2018-05-14 07:14:38.175196+00 | 2018-05-14 07:14:38.175404+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71408 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42110 | 2018-05-12 09:26:43.775727+00 | 2018-05-14 07:21:08.178871+00 | 2018-05-14 07:21:08.178871+00 | 2018-05-14 07:21:08.178875+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93175 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55394 | 2018-05-12 14:40:23.675456+00 | 2018-05-14 07:06:58.171171+00 | 2018-05-14 07:26:27.147177+00 | 2018-05-14 07:26:27.147919+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93188 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55414 | 2018-05-12 14:40:33.671126+00 | 2018-05-14 07:15:38.176369+00 | 2018-05-14 07:15:38.176369+00 | 2018-05-14 07:15:38.176374+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93200 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55434 | 2018-05-12 14:40:38.674186+00 | 2018-05-14 07:23:38.191717+00 | 2018-05-14 07:23:38.191717+00 | 2018-05-14 07:23:38.191722+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93201 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55438 | 2018-05-12 14:40:43.676002+00 | 2018-05-14 07:23:18.179621+00 | 2018-05-14 07:23:18.179621+00 | 2018-05-14 07:23:18.179626+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71450 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42170 | 2018-05-12 09:27:08.106255+00 | 2018-05-14 07:07:33.153467+00 | 2018-05-14 07:07:33.153467+00 | 2018-05-14 07:07:33.153473+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93214 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55456 | 2018-05-12 14:40:48.670159+00 | 2018-05-14 07:13:43.165456+00 | 2018-05-14 07:13:43.165456+00 | 2018-05-14 07:13:43.165468+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93215 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55460 | 2018-05-12 14:40:53.676829+00 | 2018-05-14 07:14:03.172134+00 | 2018-05-14 07:14:03.172134+00 | 2018-05-14 07:14:03.172141+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93216 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55468 | 2018-05-12 14:40:55.487214+00 | 2018-05-14 06:58:50.295551+00 | 2018-05-14 06:58:50.295569+00 | 2018-05-14 06:58:50.295571+00 | f       | active              |             |        33681 | DELETE from jsondb where path LIKE $1 and path < $2
 16385 | syndesis |  71477 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42216 | 2018-05-12 09:27:28.109477+00 | 2018-05-12 14:37:51.605173+00 | 2018-05-14 07:19:59.62134+00  | 2018-05-14 07:19:59.621721+00 | f       | active              |             |        33681 | DELETE FROM jsondb WHERE path = $1 OR path = $2 OR path = $3 OR path LIKE $4
 16385 | syndesis |  93228 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55484 | 2018-05-12 14:41:03.676553+00 | 2018-05-14 07:09:23.160613+00 | 2018-05-14 07:09:23.160613+00 | 2018-05-14 07:09:23.160626+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93241 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55502 | 2018-05-12 14:41:08.678889+00 | 2018-05-14 07:14:08.170802+00 | 2018-05-14 07:14:08.170802+00 | 2018-05-14 07:14:08.170807+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93253 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55524 | 2018-05-12 14:41:18.677208+00 | 2018-05-14 07:26:28.185719+00 | 2018-05-14 07:26:28.185719+00 | 2018-05-14 07:26:28.185723+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93254 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55528 | 2018-05-12 14:41:23.677958+00 | 2018-05-14 07:13:48.169429+00 | 2018-05-14 07:13:48.169429+00 | 2018-05-14 07:13:48.169444+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93267 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55546 | 2018-05-12 14:41:28.676404+00 | 2018-05-14 07:08:18.153714+00 | 2018-05-14 07:08:18.153714+00 | 2018-05-14 07:08:18.153718+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93268 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55550 | 2018-05-12 14:41:33.675833+00 | 2018-05-14 07:10:03.157534+00 | 2018-05-14 07:10:03.157534+00 | 2018-05-14 07:10:03.157538+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93280 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55570 | 2018-05-12 14:41:38.676968+00 | 2018-05-14 07:10:53.155514+00 | 2018-05-14 07:10:53.155514+00 | 2018-05-14 07:10:53.15552+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93281 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55574 | 2018-05-12 14:41:43.676158+00 | 2018-05-14 07:16:18.167193+00 | 2018-05-14 07:16:18.167193+00 | 2018-05-14 07:16:18.1672+00   | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93295 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55596 | 2018-05-12 14:41:53.686308+00 | 2018-05-14 07:16:48.1685+00   | 2018-05-14 07:16:48.1685+00   | 2018-05-14 07:16:48.168505+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93296 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55604 | 2018-05-12 14:41:55.484642+00 | 2018-05-14 07:14:13.167589+00 | 2018-05-14 07:14:13.167589+00 | 2018-05-14 07:14:13.167595+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93308 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55620 | 2018-05-12 14:42:03.676753+00 | 2018-05-14 07:14:33.162776+00 | 2018-05-14 07:14:33.162776+00 | 2018-05-14 07:14:33.162784+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93456 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55868 | 2018-05-12 14:43:53.687148+00 | 2018-05-14 07:18:38.173052+00 | 2018-05-14 07:18:38.173052+00 | 2018-05-14 07:18:38.173056+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93321 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55638 | 2018-05-12 14:42:08.678187+00 | 2018-05-14 07:11:53.159718+00 | 2018-05-14 07:11:53.159718+00 | 2018-05-14 07:11:53.159723+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93333 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55660 | 2018-05-12 14:42:18.676703+00 | 2018-05-14 07:20:18.180602+00 | 2018-05-14 07:20:18.180602+00 | 2018-05-14 07:20:18.180628+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93334 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55664 | 2018-05-12 14:42:23.679111+00 | 2018-05-14 07:11:13.165932+00 | 2018-05-14 07:11:13.165932+00 | 2018-05-14 07:11:13.165937+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93347 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55682 | 2018-05-12 14:42:28.684008+00 | 2018-05-14 07:11:43.164362+00 | 2018-05-14 07:11:43.164362+00 | 2018-05-14 07:11:43.165028+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93348 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55686 | 2018-05-12 14:42:33.680064+00 | 2018-05-14 07:16:58.15706+00  | 2018-05-14 07:16:58.15706+00  | 2018-05-14 07:16:58.157503+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93360 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55706 | 2018-05-12 14:42:38.682562+00 | 2018-05-14 07:16:38.170875+00 | 2018-05-14 07:16:38.170875+00 | 2018-05-14 07:16:38.17088+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93361 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55710 | 2018-05-12 14:42:43.678862+00 | 2018-05-14 07:13:33.167507+00 | 2018-05-14 07:13:33.167507+00 | 2018-05-14 07:13:33.167511+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71612 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42452 | 2018-05-12 09:29:13.122605+00 | 2018-05-14 07:13:53.174081+00 | 2018-05-14 07:13:53.174081+00 | 2018-05-14 07:13:53.174086+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93374 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55728 | 2018-05-12 14:42:48.682876+00 | 2018-05-14 07:18:53.177583+00 | 2018-05-14 07:18:53.177583+00 | 2018-05-14 07:18:53.177588+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71637 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42488 | 2018-05-12 09:29:28.114871+00 | 2018-05-14 07:22:53.182896+00 | 2018-05-14 07:22:53.182896+00 | 2018-05-14 07:22:53.1829+00   | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93375 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55732 | 2018-05-12 14:42:53.682779+00 | 2018-05-14 07:20:08.175328+00 | 2018-05-14 07:20:08.175328+00 | 2018-05-14 07:20:08.175333+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93376 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55740 | 2018-05-12 14:42:55.485451+00 | 2018-05-14 07:16:13.169465+00 | 2018-05-14 07:16:13.169465+00 | 2018-05-14 07:16:13.16947+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71651 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42522 | 2018-05-12 09:29:43.122118+00 | 2018-05-14 07:08:23.153414+00 | 2018-05-14 07:08:23.153414+00 | 2018-05-14 07:08:23.153419+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93388 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55756 | 2018-05-12 14:43:03.681197+00 | 2018-05-14 07:08:08.15404+00  | 2018-05-14 07:08:08.15404+00  | 2018-05-14 07:08:08.154045+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71666 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42552 | 2018-05-12 09:29:55.487766+00 | 2018-05-14 07:16:08.168951+00 | 2018-05-14 07:16:08.168951+00 | 2018-05-14 07:16:08.168957+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93401 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55776 | 2018-05-12 14:43:13.682443+00 | 2018-05-14 07:20:53.223661+00 | 2018-05-14 07:20:53.223661+00 | 2018-05-14 07:20:53.223666+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93413 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55796 | 2018-05-12 14:43:18.68247+00  | 2018-05-14 07:18:08.173622+00 | 2018-05-14 07:18:08.173622+00 | 2018-05-14 07:18:08.173628+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93414 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55800 | 2018-05-12 14:43:23.687476+00 | 2018-05-14 07:18:18.170396+00 | 2018-05-14 07:18:18.170396+00 | 2018-05-14 07:18:18.170401+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93428 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55818 | 2018-05-12 14:43:28.680115+00 | 2018-05-14 07:21:53.167881+00 | 2018-05-14 07:21:53.167881+00 | 2018-05-14 07:21:53.16944+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93429 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55822 | 2018-05-12 14:43:33.685411+00 | 2018-05-14 07:22:43.17985+00  | 2018-05-14 07:22:43.17985+00  | 2018-05-14 07:22:43.179853+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93441 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55842 | 2018-05-12 14:43:38.686196+00 | 2018-05-14 07:19:43.1836+00   | 2018-05-14 07:19:43.1836+00   | 2018-05-14 07:19:43.183606+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93442 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55846 | 2018-05-12 14:43:43.683791+00 | 2018-05-14 07:20:03.178207+00 | 2018-05-14 07:20:03.178207+00 | 2018-05-14 07:20:03.178211+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93455 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55864 | 2018-05-12 14:43:48.681142+00 | 2018-05-14 07:08:53.154514+00 | 2018-05-14 07:08:53.154514+00 | 2018-05-14 07:08:53.154519+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93457 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55876 | 2018-05-12 14:43:55.488009+00 | 2018-05-14 07:08:28.156143+00 | 2018-05-14 07:08:28.156143+00 | 2018-05-14 07:08:28.156148+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93469 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55892 | 2018-05-12 14:44:03.689896+00 | 2018-05-14 07:15:03.16747+00  | 2018-05-14 07:15:03.16747+00  | 2018-05-14 07:15:03.167492+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71744 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42680 | 2018-05-12 09:30:53.136214+00 | 2018-05-14 07:16:43.15909+00  | 2018-05-14 07:16:43.15909+00  | 2018-05-14 07:16:43.159101+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71745 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42688 | 2018-05-12 09:30:55.48619+00  | 2018-05-14 07:15:23.170682+00 | 2018-05-14 07:15:23.170682+00 | 2018-05-14 07:15:23.17069+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93630 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56164 | 2018-05-12 14:45:58.69309+00  | 2018-05-14 07:23:58.183551+00 | 2018-05-14 07:23:58.183551+00 | 2018-05-14 07:23:58.183556+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71771 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42726 | 2018-05-12 09:31:13.130242+00 | 2018-05-14 07:09:48.169078+00 | 2018-05-14 07:09:48.169078+00 | 2018-05-14 07:09:48.169086+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71770 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42718 | 2018-05-12 09:31:08.124909+00 | 2018-05-14 07:09:28.160249+00 | 2018-05-14 07:09:28.160249+00 | 2018-05-14 07:09:28.160254+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71784 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42750 | 2018-05-12 09:31:23.139373+00 | 2018-05-14 07:06:23.166532+00 | 2018-05-14 07:25:46.961758+00 | 2018-05-14 07:25:46.962095+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93482 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55910 | 2018-05-12 14:44:08.688462+00 | 2018-05-14 07:02:14.450414+00 | 2018-05-14 07:02:14.509878+00 | 2018-05-14 07:02:14.50988+00  | f       | active              |             |        33681 | DELETE from jsondb where path LIKE $1 and path < $2
 16385 | syndesis |  71798 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42772 | 2018-05-12 09:31:33.134195+00 | 2018-05-14 07:08:13.15549+00  | 2018-05-14 07:08:13.15549+00  | 2018-05-14 07:08:13.155495+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93483 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55914 | 2018-05-12 14:44:13.697003+00 | 2018-05-14 07:19:48.177036+00 | 2018-05-14 07:19:48.177036+00 | 2018-05-14 07:19:48.177041+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93495 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55934 | 2018-05-12 14:44:18.686211+00 | 2018-05-14 07:06:53.162295+00 | 2018-05-14 07:26:15.987927+00 | 2018-05-14 07:26:15.988133+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93496 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55938 | 2018-05-12 14:44:23.688966+00 | 2018-05-14 07:13:08.165396+00 | 2018-05-14 07:13:08.165396+00 | 2018-05-14 07:13:08.165561+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93509 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55956 | 2018-05-12 14:44:28.684477+00 | 2018-05-14 07:12:18.163516+00 | 2018-05-14 07:12:18.163516+00 | 2018-05-14 07:12:18.16371+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93510 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55960 | 2018-05-12 14:44:33.693276+00 |                               | 2018-05-14 07:24:21.135709+00 | 2018-05-14 07:26:43.340698+00 | f       | idle                |             |              | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93522 |    16384 | syndesis |                  | 10.128.0.1  |                 |       55980 | 2018-05-12 14:44:38.688091+00 | 2018-05-14 07:23:13.187517+00 | 2018-05-14 07:23:13.187517+00 | 2018-05-14 07:23:13.187523+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93535 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56002 | 2018-05-12 14:44:48.68711+00  | 2018-05-14 07:13:38.16621+00  | 2018-05-14 07:13:38.16621+00  | 2018-05-14 07:13:38.166215+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93536 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56006 | 2018-05-12 14:44:53.70355+00  | 2018-05-14 07:19:38.175546+00 | 2018-05-14 07:19:38.175546+00 | 2018-05-14 07:19:38.175559+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93537 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56014 | 2018-05-12 14:44:55.484971+00 | 2018-05-14 07:08:48.156922+00 | 2018-05-14 07:08:48.156922+00 | 2018-05-14 07:08:48.156927+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93549 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56030 | 2018-05-12 14:45:03.68793+00  | 2018-05-14 07:20:28.177788+00 | 2018-05-14 07:20:28.177788+00 | 2018-05-14 07:20:28.177792+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93562 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56048 | 2018-05-12 14:45:08.697542+00 | 2018-05-14 07:22:28.177697+00 | 2018-05-14 07:22:28.177697+00 | 2018-05-14 07:22:28.177709+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93563 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56052 | 2018-05-12 14:45:13.692739+00 | 2018-05-14 07:11:38.159217+00 | 2018-05-14 07:11:38.159217+00 | 2018-05-14 07:11:38.159221+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93576 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56072 | 2018-05-12 14:45:18.693845+00 | 2018-05-14 07:26:33.188374+00 | 2018-05-14 07:26:33.188374+00 | 2018-05-14 07:26:33.188379+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93577 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56076 | 2018-05-12 14:45:23.692429+00 | 2018-05-14 07:19:53.174129+00 | 2018-05-14 07:19:53.174129+00 | 2018-05-14 07:19:53.174133+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93590 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56094 | 2018-05-12 14:45:28.689421+00 | 2018-05-14 07:23:23.181874+00 | 2018-05-14 07:23:23.181874+00 | 2018-05-14 07:23:23.181879+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93591 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56098 | 2018-05-12 14:45:33.686751+00 | 2018-05-14 07:08:33.151598+00 | 2018-05-14 07:08:33.151598+00 | 2018-05-14 07:08:33.151602+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93603 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56120 | 2018-05-12 14:45:43.69644+00  | 2018-05-14 07:17:23.169909+00 | 2018-05-14 07:17:23.169909+00 | 2018-05-14 07:17:23.169927+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71944 |    16384 | syndesis |                  | 10.128.0.1  |                 |       43022 | 2018-05-12 09:33:23.256116+00 | 2018-05-14 07:22:08.184813+00 | 2018-05-14 07:22:08.184813+00 | 2018-05-14 07:22:08.184818+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71918 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42972 | 2018-05-12 09:32:58.14813+00  | 2018-05-14 07:13:03.165143+00 | 2018-05-14 07:13:03.165143+00 | 2018-05-14 07:13:03.165148+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71931 |    16384 | syndesis |                  | 10.128.0.1  |                 |       42992 | 2018-05-12 09:33:08.134654+00 | 2018-05-14 07:07:03.148709+00 | 2018-05-14 07:26:30.737038+00 | 2018-05-14 07:26:30.737256+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93616 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56138 | 2018-05-12 14:45:48.690016+00 | 2018-05-14 07:17:33.172361+00 | 2018-05-14 07:17:33.172361+00 | 2018-05-14 07:17:33.172366+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93617 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56142 | 2018-05-12 14:45:53.698249+00 | 2018-05-14 07:12:48.164198+00 | 2018-05-14 07:12:48.164198+00 | 2018-05-14 07:12:48.164202+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  93618 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56150 | 2018-05-12 14:45:55.485118+00 | 2018-05-14 07:20:13.178035+00 | 2018-05-14 07:20:13.178035+00 | 2018-05-14 07:20:13.17804+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |    960 |       10 | postgres | psql             |             |                 |          -1 | 2018-05-14 07:26:45.445884+00 | 2018-05-14 07:26:45.811417+00 | 2018-05-14 07:26:45.811417+00 | 2018-05-14 07:26:45.811532+00 | f       | active              |             |        33681 | select * from pg_stat_activity;
 16385 | syndesis |  93643 |    16384 | syndesis |                  | 10.128.0.1  |                 |       56184 | 2018-05-12 14:46:08.712397+00 | 2018-05-14 07:22:03.178794+00 | 2018-05-14 07:22:03.178794+00 | 2018-05-14 07:22:03.178799+00 | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |  71985 |    16384 | syndesis |                  | 10.128.0.1  |                 |       43090 | 2018-05-12 09:33:53.139066+00 | 2018-05-14 07:09:03.153954+00 | 2018-05-14 07:09:03.153954+00 | 2018-05-14 07:09:03.15396+00  | f       | active              |             |        33681 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 16385 | syndesis |    946 |       10 | postgres |                  |             |                 |             | 2018-05-14 07:26:29.666334+00 | 2018-05-14 07:26:30.264195+00 | 2018-05-14 07:26:30.264195+00 | 2018-05-14 07:26:30.264205+00 | f       | active              |             |        33681 | autovacuum: VACUUM public.jsondb
(99 rows)

maschmid commented 6 years ago

Server logs and jstack on http://xtf.cz/syndesis.io-2607.tar.gz

maschmid commented 6 years ago

Note that there are 98 threads with the name "Logs Controller" in it, which is very suspicious.

paoloantinori commented 6 years ago

@chirino @zregvart

I'm not familiar with org.skife.jdbi.v2.DBI usage patterns, but I have notice this that looks like a smell:

In io.syndesis.server.jsondb.impl.SqlJsonDB#getAsStreamingOutput we invoke:

final Handle h = dbi.open();

And then we have a set of calls to .close() in multiple finally blocks.

But most of them are in a conditional statement:

                // if we are producing results, then defer closing the iterator
                if (result == null) {
                    iterator.close();
                }

And even the more broader one has that check.

As far as I can see, a reference to the local variable Handle h is never passed to context that are external to the current one. So at least, at static level checking, I think there is the possibility that that h instance never recevies a close() call.

Am I reading this right? Can this leak explain the issue that has been reported?

chirino commented 6 years ago

@maschmid agree, 98 thread of "Logs Controller" is wrong. I assume you had way fewer integrations right?

maschmid commented 6 years ago

@chirino I have 5 (3x 1s timer -> amq queue, 2x amq queue -> log )

rhuss commented 6 years ago

From the IRC discussion:

The scheduleWithFixedDelay only starts a thread with the executor, it doesnt do the actual polling. so starting that thread is very quick of course and 5 seconds later the next thread is started. https://github.com/rhuss/syndesis/blob/b9bb7adb3c5b0307a5589d4c8c7852929046f18b/app/server/logging/jsondb/src/main/java/io/syndesis/server/logging/jsondb/controller/ActivityTrackingController.java#L95 you should run the poll with scheduleWithFixedDelay to guarantee that it's running serial.

rhuss commented 6 years ago

We fixed now the database contention issue, but I'm still afraid of stressing the DB too much with polling the pods if this takes longer than 5s as we have now this situation in the polling thread:

[poll > 5s ], [wait 5s], [poll > 5s], ...

so at least half of the time (but probably even more), jsondb is busy with doing the select. As I suspect that the duration of the query increases substantially with the numbers of activities logged (not sure here), the situation will get even worse over time.

Nevertheless I'm afraid that this kind of handling the activities will also severely impact the user experience when using the UI because of delays in fetching the backend data. But I already mentioned that several times and don't want to repeat myself too much :)

We should put 'considering separate log backend from ui backend' as tech-debt story probably.

chirino commented 6 years ago

agree.. we should likely move this activity stuff to it's own DB tables for 7.1 as a tech debt item.

apupier commented 6 years ago

the related build pointed on IRC has failed: https://ci.fabric8.io/view/release%20builds/job/syndesis-release/113/console

18:22:26 [ERROR] Errors: 18:22:26 [ERROR] OutMessageCaptureProcessorTest.testCaptureWithSplitAndSchedule:291 NullPointer

apupier commented 6 years ago

the PR build was green https://circleci.com/gh/syndesisio/syndesis/31464

is the OutMessageCaptureProcessorTest.testCaptureWithSplitAndSchedule a known unstable test? (seems not as I found no references in the github issues) Is the release job launching a different set of test than the PR one?

EDIT: the job has been relaunched and the test passed, so seems to be an unstable one EDIT2: I reported the unstable behavior https://github.com/syndesisio/syndesis/issues/2658

rhuss commented 6 years ago

@dsimansk just noted, that there a 2 hours of activities missing in the logs of his test with 16 integrations.

From what I see from the stacktrace it looks like this:

There are 16 monitor threads picking up the logs from the pods and store them into a 1000 sized LinkedBlockingDeque.
There is 1 thread for maintenance (checking pods status and cleaning up)
There is 1 writer thread picking up the activities from the LinkedBlockingDeque, writing them into the DB.

From the stacktrace you see that all 16 monitor threads are blocked while writing because the queue is full. You see the 1 writer thread running under full blast, but he can not empty the queue fast enough. This already causes a delay of 2 hours (after running how long, @dsimansk ?), and also putting back pressure on the OpenShift API server who want to stream the log output (but is blocked).

This is clearly a limitation of write performance of jsondb and unfortunately I don't see an easy solution except of exchange the backend store.

It's this line which is just to slow to cope with 16 integrations running in parallel.

@chirino is this a correct analysis ?

so, what do we do now ? Just dropping entries would reduce the back pressure at least. I think it can not be assumed that the pressure will be reduced over time, calming down the situation as the integrations are continue to run periodically (if based on a timer).

@dsimansk btw, with which rate are the activity entries are produced ?

rhuss commented 6 years ago

As this problem scale with the number of integrations (and of course with the nature of the integration, if its only generating events once an hour its obviously not an issue), we can at least say that there is an upper limit of parallel integrations we support per installation. Might be good with 5 integrations.

At least we dont have a DB connection congestion anymore.

dsimansk commented 6 years ago

The rate is exchange per 1s. The last entry is logged & displayed after ~4h of uptime from what I've seen.

Yes it seems that DB connections are fine at least and server is able query DB.

@rhuss should I redeploy with lower number of parallel Integrations, e.g. 5 or maybe 8 to let it run over-night?

rhuss commented 6 years ago

Another optimization we could try is to throttle on the LogMonitor side, so that they collect activities for e.g. 5 minutes and then put into the queue where the writer thread could do an bulk update which might potentially be faster (?).

Drawback would be that the UI would have also a delay when showing the activities (but probably much less than 2 hours)

rhuss commented 6 years ago

@dsimansk yes, would be probably better to run with less parallel integrations, maybe even with a slower rate (8 integrations with 5s ?)

rhuss commented 6 years ago

Its really a trial & error thingy at the moment to find the proper limits.

dsimansk commented 6 years ago

@rhuss @chirino Lowering number of parallel integrations to 8 and change to 5s delay proceeded further to ~9h of proper tracking then delay started to increase slowly. Should I further drop to 5 parallel and keep the delay at 5s?

rhuss commented 6 years ago

@dsimansk yes, you can try this. Actually only the number "nr. integrations / delay" should be important. So we started with 16/1 = 16 runs per second (delay starting after 4 hours), now it was 8/5 = 1.6 runs per second (which is 10 times less, delay starting after 9 hours). Assuming a linear relationship (which it is not, but lets start with this as in any series approximation), we have the following equation

duration = m * rate + b
m = (9h - 4h) / (16 - 1.6) = 0.35 h
b = (duration - 0.34 * rate) = 9h - 0.35 h * 16 = 3.4h

==> duration = 0.35h * rate + 3.4h

That means for a rate of 1 the delay should occur around 3.7 hours. As you see, its an oversimplication because that means that for a rate of 0 we would see starting to see a delay at 3.4 hours. So I expect to have a higher number and if we could collect more data we can get an even better feeling (actually I expect to have no delays at all if we get that ratio low enough).

tl;dr - Try 5 integrations with a 5 s frequency and lets see how it is going. I wonder though whether we could find the point in time where the delay starts more accurately (e.g. by looping with a select over the activity table and checking the last entry programmatically).

chirino commented 6 years ago

@dsimansk any changes with the latest nightly?

dsimansk commented 6 years ago

@chirino it's definitely better. 5 integrations with 5s delay: after 24h Activity tracking is consistent and nothing missing in sequence

Therefor I've decided to increase the load to establish new maximum. But deploying 8 integrations with 1s delay is still too much, after 12h of runtime the missing entries start to occur.

Wdyt about the possible max to test here?

mmelko commented 6 years ago

I've created https://github.com/syndesisio/syndesis/issues/3350 as follow up for this.

syndesisio / syndesis