search

wso2 / product-is

Welcome to the WSO2 Identity Server source code! For info on working with the WSO2 Identity Server repository and contributing code, click the link below.
http://wso2.github.io/
Apache License 2.0
748 stars 730 forks source link

PoolExhaustedException found on large concurrency #6823

Closed aaujayasena closed 4 weeks ago

aaujayasena commented 5 years ago

product version: 5.8.0 /5.6.0 Severity: medium Priority: high

Exception, 'PoolExhaustedException' found in the performance test execution for 500 concurrency in below scenarios.

DB: mysql OS: ubuntu

TID: [-1234] [] [2019-11-07 03:46:46,399] ERROR {org.wso2.carbon.identity.application.authentication.framework.store.SessionDataStore} -  org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-9443-exec-358] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none available[size:300; busy:300; idle:0; lastwait:59999]. 
org.wso2.carbon.identity.base.IdentityRuntimeException: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-9443-exec-358] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none available[size:300; busy:300; idle:0; lastwait:59999].
    at org.wso2.carbon.identity.base.IdentityRuntimeException.error(IdentityRuntimeException.java:71)
    at org.wso2.carbon.identity.core.persistence.JDBCPersistenceManager.getDBConnection(JDBCPersistenceManager.java:145)
    at org.wso2.carbon.identity.core.util.IdentityDatabaseUtil.getDBConnection(IdentityDatabaseUtil.java:47)
    at org.wso2.carbon.identity.application.authentication.framework.store.SessionDataStore.persistSessionData(SessionDataStore.java:508)
    at org.wso2.carbon.identity.application.authentication.framework.store.SessionDataStore.storeSessionData(SessionDataStore.java:361)
    at org.wso2.carbon.identity.application.authentication.framework.store.SessionDataStore.storeSessionData(SessionDataStore.java:350)
    at org.wso2.carbon.identity.oauth.cache.SessionDataCache.addToCache(SessionDataCache.java:60)
    at org.wso2.carbon.identity.oauth.endpoint.util.EndpointUtil.getUserConsentURL(EndpointUtil.java:539)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.getUserConsentURL(OAuth2AuthzEndpoint.java:2151)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.getUserConsentURL(OAuth2AuthzEndpoint.java:2160)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.promptUserForConsent(OAuth2AuthzEndpoint.java:1790)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.handleConsent(OAuth2AuthzEndpoint.java:1753)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.doUserAuthorization(OAuth2AuthzEndpoint.java:1725)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.handleSuccessfulAuthentication(OAuth2AuthzEndpoint.java:650)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.handleAuthenticationResponse(OAuth2AuthzEndpoint.java:620)
    at org.wso2.carbon.identity.oauth.endpoint.authz.OAuth2AuthzEndpoint.authorize(OAuth2AuthzEndpoint.java:223)
    at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:225)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.wso2.carbon.webapp.mgt.filter.AuthorizationHeaderFilter.doFilter(AuthorizationHeaderFilter.java:85)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
    at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
    at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
    at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:65)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
    at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
    at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1780)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1739)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-9443-exec-358] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none available[size:300; busy:300; idle:0; lastwait:59999].
    at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:703)
    at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:200)
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
    at org.wso2.carbon.identity.core.persistence.JDBCPersistenceManager.getDBConnection(JDBCPersistenceManager.java:128)
    ... 74 more

Issue is linked with issue [1]

[1] https://github.com/wso2/product-is/issues/1830 [2] https://stackoverflow.com/questions/29368032/tomcat-connection-pool-exhausted

omindu commented 5 years ago

@aaujayasena Can this be reproduced always ? If so we can try to drop long running connections and see whether there's a connection leak

aaujayasena commented 5 years ago

@omindu This is a intermittent issue and error reported percentage is nearly 0.04% . Heap size: 2G No of cores : 4 cores Concurrency users: 500 So your proposing to execute long running test with 500 concurrency or create a peak time of 500 concurrency in the middle of long running execution?

omindu commented 5 years ago

@aaujayasena I'm proposing to execute long running test with 500 concurrency and check for connection leaks.

Also, what's the maxActive connection count for identity data source ?

aaujayasena commented 5 years ago

@omindu maxActive connection count is 300

omindu commented 5 years ago

We should increase the connection count for IDN DB to 500

isharak commented 4 weeks ago

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!