Closed ops4j-issues closed 8 years ago
Grzegorz Grzybek commented
Here's how it happens...
PID for cxf-rt-transport-http bundle is changed and CXF registers some servlet:
"CM Configuration Updater (ManagedService Update: pid=[org.apache.cxf.osgi])@3043" daemon prio=5 tid=0x16 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at org.eclipse.jetty.server.session.HashSessionManager.doStart(HashSessionManager.java:135)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x1a0f> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
at org.eclipse.jetty.server.session.SessionHandler.doStart(SessionHandler.java:116)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x1a10> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:784)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.startContext(HttpServiceContext.java:603)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doStart(HttpServiceContext.java:261)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x1a11> (a java.lang.Object)
at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$1.start(JettyServerImpl.java:279)
at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:221)
at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:195)
at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:179)
- locked <0x1a12> (a java.lang.Object)
at org.ops4j.pax.web.service.internal.HttpServiceProxy.registerServlet(HttpServiceProxy.java:64)
at org.apache.cxf.transport.http.osgi.ServletExporter.updated(ServletExporter.java:108)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updated(ManagedServiceTracker.java:189)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updateService(ManagedServiceTracker.java:152)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.provideConfiguration(ManagedServiceTracker.java:85)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.provide(ConfigurationManager.java:1440)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.run(ConfigurationManager.java:1396)
at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
at java.lang.Thread.run(Thread.java:745)
org.ops4j.pax.web.service.jetty.internal.JettyServerImpl#getContext()
returns a Lifecycle which, when started, sets org.eclipse.jetty.server.handler.ContextHandler#_classLoader
to org.ops4j.pax.web.service.spi.util.ResourceDelegatingBundleClassLoader
org.eclipse.jetty.server.handler.ContextHandler#doStart()
is called (this.getClass() == org.ops4j.pax.web.service.jetty.internal.HttpServiceContext), it set's the above CL as TCCL, starting of Jetty beans continuesorg.eclipse.jetty.server.session.HashSessionManager#doStart
is called and _timer=new ScheduledExecutorScheduler(toString()+"Timer",true)
is initialized, this sets org.eclipse.jetty.util.thread.ScheduledExecutorScheduler#classloader
to the above TCCLorg.eclipse.jetty.server.session.HashSessionManager#setScavengePeriod()
is called which schedules scavenger task"CM Configuration Updater (ManagedService Update: pid=[org.apache.cxf.osgi])@3043" daemon prio=5 tid=0x16 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at org.eclipse.jetty.util.thread.ScheduledExecutorScheduler$1.newThread(ScheduledExecutorScheduler.java:74)
at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:612)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1587)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.eclipse.jetty.util.thread.ScheduledExecutorScheduler.schedule(ScheduledExecutorScheduler.java:104)
at org.eclipse.jetty.server.session.HashSessionManager.setScavengePeriod(HashSessionManager.java:322)
- locked <0x171f> (a org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager)
at org.eclipse.jetty.server.session.HashSessionManager.doStart(HashSessionManager.java:155)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x1a45> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
at org.eclipse.jetty.server.session.SessionHandler.doStart(SessionHandler.java:116)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x1a42> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:784)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.startContext(HttpServiceContext.java:603)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doStart(HttpServiceContext.java:261)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x1a11> (a java.lang.Object)
at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$1.start(JettyServerImpl.java:279)
at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:221)
at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:195)
at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:179)
- locked <0x1ab8> (a java.lang.Object)
at org.ops4j.pax.web.service.internal.HttpServiceProxy.registerServlet(HttpServiceProxy.java:64)
at org.apache.cxf.transport.http.osgi.ServletExporter.updated(ServletExporter.java:108)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updated(ManagedServiceTracker.java:189)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updateService(ManagedServiceTracker.java:152)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.provideConfiguration(ManagedServiceTracker.java:85)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.provide(ConfigurationManager.java:1440)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.run(ConfigurationManager.java:1396)
at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
at java.lang.Thread.run(Thread.java:745)
this sets TCCL of thread from the scheduler to current TCCL which is org.ops4j.pax.web.service.spi.util.ResourceDelegatingBundleClassLoader
.
Grzegorz Grzybek commented
Fixed in web-4.2.x: https://github.com/ops4j/org.ops4j.pax.web/commit/e23acea613434378cdd450636308760d802b5968
Fixed in master: https://github.com/ops4j/org.ops4j.pax.web/commit/b6ec897a0d94afcf97b8575fbfd4b8ebe7f63fcf
Grzegorz Grzybek created PAXWEB-1010
I'm investigating memory dumps of Karaf/Fuse applications after several redeployments of bundles/features.
I focused on finding these bundle revisions which have more than one active bundle wiring (Felix implementation) and I was looking for GC roots of these wirings (connected with classloaders). I also was checking why there are more than one active instance of
org.eclipse.jetty.server.ServerConnector
and I saw this:the problem is that if
org.ops4j.pax.web
PID was updated, pax-web-jetty services (like ServerController or HttpService) were republished, but running web bundles were not fully unpublished (bug coming soon - related to wrong ref counting in pax-web-extender-war). In the end, running HashSessionManager related threads had TCCL set toorg.ops4j.pax.web.service.spi.util.ResourceDelegatingBundleClassLoader
related to not-fully-unpublished WARs and the chain of references goes from bundle, through activator, server controller to ServerConnector.What's more, this (and this pax-web-extender-war bug) leads to (randomly)
java.lang.LinkageError: org/ops4j/pax/web/service/spi/model/ServerModel
because we had more than one connector related to different wirings. Unpublished webapps were still available inorg.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection#_handlers
.I have a fix for this issue, which is part of resolving classloader related problems in pax-web.
Affects: 4.2.8 Fixed in: 4.3.0, 6.0.0 Votes: 0, Watches: 1