search

javaee / jersey

This is no longer the active Jersey repository. Please see the README.md
http://jersey.github.io
Other
2.87k stars 2.36k forks source link

Jersey Threads blocking under heavy load #3474

Open glassfishrobot opened 7 years ago

glassfishrobot commented 7 years ago

Been constantly getting the below error in weblogic when we put JERSEY under heavy stress.

####<Dec 7, 2016 3:09:32 PM GMT> <Info> <Health> <prlprws01> <PayRoll_Srv1> <weblogic.GCMonitor> <<anonymous>> <> <> <1481123372881> <BEA-310002> <48% of the total memory in the server is free.> 
####<Dec 7, 2016 3:12:21 PM GMT> <Error> <WebLogicServer> <prlprws01> <PayRoll_Srv1> <[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1481123541709> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "645" seconds working on the request "Workmanager: default, Version: 5, Scheduled=true, Started=true, Started time: 645131 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:198)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:846)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1006)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1315)
    jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
    jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    org.glassfish.jersey.servlet.internal.ResponseWriter.getResponseContext(ResponseWriter.java:277)
    org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:191)
    org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:413)
    org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:750)
    org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:424)
    org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:414)
    org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:311)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)
    org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)
    org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:403)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:334)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:221)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3284)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3254)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
    weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
    weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2163)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2089)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2074)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1513)
    weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Environment

Production

Affected Versions

[2.19]

glassfishrobot commented 7 years ago

Reported by ahm_mushtaq

glassfishrobot commented 7 years ago

@pavelbucek said: This is impossible to diagnose without a proper context or a reproducer.

Can you try to re-create this with minimal Jersey/JAX-RS application and share it somewhere?

glassfishrobot commented 7 years ago

ahm_mushtaq said: Sure will try doing it..any hints on what could possibly be causing this error? Its quite intermittent

glassfishrobot commented 7 years ago

This issue was imported from java.net JIRA JERSEY-3202