search

opendevstack / ods-provisioning-app

Provisioning app, which triggers project and component provisions (including Jira / Confluence / BitBucket and OCP resource creation)
Apache License 2.0
15 stars 20 forks source link

First Time login into Provision App takes very long #106

Closed clemensutschig closed 5 years ago

clemensutschig commented 5 years ago

looks like there is a massive Performance issue with the crowd integration api fetching the cache - depending on the overall group size

"http-nio-8080-exec-10" #36 daemon prio=5 os_prio=0 tid=0x000000001e0b1000 nid=0x33b4 runnable [0x0000000024ccc000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
        - locked <0x00000000deb44970> (a java.lang.Object)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        - locked <0x00000000deb45000> (a sun.security.ssl.AppInputStream)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        - locked <0x00000000deb4e0a0> (a java.io.BufferedInputStream)
        at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
        at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
        at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        at org.codehaus.xfire.transport.http.CommonsHttpMessageSender.send(CommonsHttpMessageSender.java:369)
        at org.codehaus.xfire.transport.http.HttpChannel.sendViaClient(HttpChannel.java:123)
        at org.codehaus.xfire.transport.http.HttpChannel.send(HttpChannel.java:48)
        at org.codehaus.xfire.handler.OutMessageSender.invoke(OutMessageSender.java:26)
        at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
        at org.codehaus.xfire.client.Invocation.invoke(Invocation.java:79)
        at org.codehaus.xfire.client.Invocation.invoke(Invocation.java:114)
        at org.codehaus.xfire.client.Client.invoke(Client.java:336)
        at org.codehaus.xfire.client.XFireProxy.handleRequest(XFireProxy.java:77)
        at org.codehaus.xfire.client.XFireProxy.invoke(XFireProxy.java:57)
        at com.sun.proxy.$Proxy107.findAllGroupRelationships(Unknown Source)
        at com.atlassian.crowd.service.soap.client.SecurityServerClientImpl.findAllGroupRelationships(SecurityServerClientImpl.java:1332)
        at com.atlassian.crowd.service.cache.CachingGroupManager$Server.getAllGroupRelationships(CachingGroupManager.java:191)
        at com.atlassian.crowd.service.cache.CachingGroupManager.getAllGroupNames(CachingGroupManager.java:134)
        - locked <0x00000000842ccb40> (a java.lang.Object)
        at com.atlassian.crowd.service.cache.CachingGroupMembershipManager.getAncestorsForGroups(CachingGroupMembershipManager.java:223)
        at com.atlassian.crowd.service.cache.CachingGroupMembershipManager.getMemberships(CachingGroupMembershipManager.java:124)
        at com.atlassian.crowd.integration.springsecurity.user.CrowdUserDetailsServiceImpl.getAuthorities(CrowdUserDetailsServiceImpl.java:135)
        at com.atlassian.crowd.integration.springsecurity.user.CrowdUserDetailsServiceImpl.loadUserByToken(CrowdUserDetailsServiceImpl.java:73)
        at com.atlassian.crowd.integration.springsecurity.RemoteCrowdAuthenticationProvider.loadUserByToken(RemoteCrowdAuthenticationProvider.java:62)
        at com.atlassian.crowd.integration.springsecurity.CrowdAuthenticationProvider.authenticateCrowdSSO(CrowdAuthenticationProvider.java:257)
        at com.atlassian.crowd.integration.springsecurity.CrowdAuthenticationProvider.authenticate(CrowdAuthenticationProvider.java:110)
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
        at com.atlassian.crowd.integration.springsecurity.CrowdSSOAuthenticationProcessingFilter.requiresAuthentication(CrowdSSOAuthenticationProcessingFilter.java:77)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        - locked <0x00000000ee01c458> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

The internal call of getAuthorities - goes to the caching manager

com.atlassian.crowd.service.cache.CachingGroupMembershipManager.getAncestorsForGroups(CachingGroupMembershipManager.java:223)
        at com.atlassian.crowd.service.cache.CachingGroupMembershipManager.getMemberships(CachingGroupMembershipManager.java:124)
        at com.atlassian.crowd.integration.springsecurity.user.CrowdUserDetailsServiceImpl.getAuthorities(CrowdUserDetailsServiceImpl.java:135)
        at com.atlassian.crowd.integration.springsecurity.user.CrowdUserDetailsServiceImpl.loadUserByToken(CrowdUserDetailsServiceImpl.java:73)

to end up in ... 

        at com.sun.proxy.$Proxy107.findAllGroupRelationships(Unknown Source)
        at com.atlassian.crowd.service.soap.client.SecurityServerClientImpl.findAllGroupRelationships(SecurityServerClientImpl.java:1332)
        at com.atlassian.crowd.service.cache.CachingGroupManager$Server.getAllGroupRelationships(CachingGroupManager.java:191)
clemensutschig commented 5 years ago

@stitakis i played with this today.. And the api is really crappy.. the only Solution i could come up with is to do this leading in Startup... 2 minutes..

clemensutschig commented 5 years ago

ok - we found a workaround - it's create a second crowd directory with a small set of groups only (based on https://github.com/opendevstack/ods-provisioning-app/blob/master/src/main/resources/application.properties#L105-L106)