search

CandyShop / gerrit

Automatically exported from code.google.com/p/gerrit
Apache License 2.0
1 stars 0 forks source link

Create Project is blocking on replication plugin #3412

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Affected Version: 2.10.2

What steps will reproduce the problem?
1. Have a large replication queue to any Gerrit slave
2. Attempt to create a project over the REST API

What is the expected output? What do you see instead?
Expect project to create normally, but instead, the request times out at the 
reverse proxy.

Please provide any additional information below.
I captured the following jstack output:
"HTTP-1366923" #1366923 prio=5 os_prio=0 tid=0x00007fbd82e40000 nid=0x42ea 
waiting for monitor entry [0x00007fbd4bffc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:93)
        - waiting to lock <0x00007fc1d5e45300> (a com.google.gerrit.server.git.LocalDiskRepositoryManager$Lifecycle$1)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.connect(ReplicationQueue.java:416)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.executeRemoteSsh(ReplicationQueue.java:396)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.createRemoteSsh(ReplicationQueue.java:277)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.createProject(ReplicationQueue.java:238)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.onNewProjectCreated(ReplicationQueue.java:151)
        at com.google.gerrit.server.project.PerformCreateProject.createProject(PerformCreateProject.java:130)
        at com.google.gerrit.server.project.CreateProject.apply(CreateProject.java:142)
        at com.google.gerrit.server.project.CreateProject.apply(CreateProject.java:51)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:304)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.RequireIdentifiedUserFilter.doFilter(RequireIdentifiedUserFilter.java:55)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.ProjectDigestFilter.doFilter(ProjectDigestFilter.java:107)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        - locked <0x00007fc2db8a9718> (a com.google.inject.servlet.GuiceFilter$Context)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1636)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:564)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:498)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:98)
        at org.eclipse.jetty.server.Server.handle(Server.java:461)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:284)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

Since there's a long replication queue ahead of the creation, the creation 
times out.  This would better be handled if projects were created 
asynchronously/queued for replication on the mirrors, instead of requiring 
synchronous creation.

Original issue reported on code.google.com by dougk....@gmail.com on 5 Jun 2015 at 12:27

GoogleCodeExporter commented 9 years ago 
Actually, it appears the original assumption of the replication queue was 
incorrect.  The issue was most likely caused by a slave going offline during 
replication (really, became unresponsive and required a restart).  Since the 
connection was held open and never reset, the only way to resolve was restart 
Gerrit.

Original comment by dougk....@gmail.com on 5 Jun 2015 at 12:35