search

wso2 / api-manager

All issues, tasks, improvements and new features of WSO2 API Manager
Apache License 2.0
34 stars 8 forks source link

Recurring Unresponsiveness of a thread that handle the internal V1 API during CP Node Startup #2855

Closed cbabey closed 6 months ago

cbabey commented 7 months ago

Description

when starting the CP nodes, we noticed a recurring issue where the thread responsible for handling internal Rest API [1] became unresponsive, remaining stuck for over 10 minutes. Consequently, the thread is eventually removed by the CarbonStuckThreadDetectionValve.

TID: [] [] [2024-04-24 17:31:08,429]  WARN {org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve} - Thread [https-jsse-nio-9443-exec-42] (id=[{6}]) has been active for [609,293] milliseconds  to serve the same request for [https://localhost:9443l/internal/data/v1/apis?gatewayLabel=Production+and+Sandbox] and may be stuck (configured threshold for this StuckThreadDetectionValve is [600] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck., tenantDomain=carbon.super 7 java.lang.Throwable
    at org.wso2.carbon.user.core.authorization.PermissionTree.updatePermissionTree(PermissionTree.java:1003)
    at org.wso2.carbon.user.core.authorization.PermissionTree.updatePermissionTree(PermissionTree.java:961)
    at org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager.isRoleAuthorized(JDBCAuthorizationManager.java:198)
    at jdk.internal.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
    at java.base@11.0.23/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.23/java.lang.reflect.Method.invoke(Method.java:566)
    at org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager$2.run(JDBCAuthorizationManager.java:1518)
    at java.base@11.0.23/java.security.AccessController.doPrivileged(Native Method)
    at org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager.callSecure(JDBCAuthorizationManager.java:1515)
    at org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager.isRoleAuthorized(JDBCAuthorizationManager.java:187)
    at org.wso2.carbon.apimgt.persistence.utils.RegistryPersistenceUtil.loadloadTenantAPIRXT(RegistryPersistenceUtil.java:496)
    at org.wso2.carbon.apimgt.persistence.RegistryPersistenceImpl.getRegistry(RegistryPersistenceImpl.java:3140)
    at org.wso2.carbon.apimgt.persistence.RegistryPersistenceImpl.getSecuritySchemeOfAPI(RegistryPersistenceImpl.java:655)
    at org.wso2.carbon.apimgt.impl.APIProviderImpl.getSecuritySchemeOfAPI_aroundBody388(APIProviderImpl.java:4996)
    at org.wso2.carbon.apimgt.impl.APIProviderImpl.getSecuritySchemeOfAPI(APIProviderImpl.java:1)
    at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.getSecuritySchemeOfAPI(UserAwareAPIProvider.java:1)
    at org.wso2.carbon.apimgt.internal.service.utils.SubscriptionValidationDataUtil.fromAPItoDTO(SubscriptionValidationDataUtil.java:96)
    at org.wso2.carbon.apimgt.internal.service.utils.SubscriptionValidationDataUtil.fromAPIListToAPIListDTO(SubscriptionValidationDataUtil.java:157)
    at org.wso2.carbon.apimgt.internal.service.impl.ApisApiServiceImpl.apisGet(ApisApiServiceImpl.java:78)
    at org.wso2.carbon.apimgt.internal.service.ApisApi.apisGet(ApisApi.java:51)
    at java.base@11.0.23/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base@11.0.23/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base@11.0.23/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.23/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:279)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
    at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:119)
    at org.wso2.carbon.identity.context.rewrite.valve.OrganizationContextRewriteValve.invoke(OrganizationContextRewriteValve.java:116)
    at org.wso2.carbon.tomcat.ext.valves.SameSiteCookieValve.invoke(SameSiteCookieValve.java:38)
    at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:165)
    at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:118)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:106)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:67)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:152)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:63)
    at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:137)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base@11.0.23/java.lang.Thread.run(Thread.java:834)

[1] [[https://localhost:9443l/internal/data/v1/apis?gatewayLabel=Production+and+Sandbox]

Steps to Reproduce

This problem arises when the query returns over 50k entries and there are over 2k published APIs.

SELECT UM_ROLE_NAME, UM_RESOURCE_ID, UM_IS_ALLOWED, UM_ACTION, UM_DOMAIN_NAME FROM UM_PERMISSION, UM_ROLE_PERMISSION, UM_DOMAIN WHERE UM_ROLE_PERMISSION.UM_PERMISSION_ID=UM_PERMISSION.UM_ID AND UM_ROLE_PERMISSION.UM_DOMAIN_ID=UM_DOMAIN.UM_DOMAIN_ID AND UM_PERMISSION.UM_TENANT_ID=? AND UM_ROLE_PERMISSION.UM_TENANT_ID=?

Affected Component

APIM

Version

4.2.0

Environment Details (with versions)

No response

Relevant Log Output

No response

Related Issues

No response

Suggested Labels

No response

chashikajw commented 6 months ago

Closing the issue since master PR is merged.