We've been experience a deadlock in classloading when using Wildfly. Full details of deadlock below.
It seems in org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl There is a mix of synchronize, global locking and also locking within the default Java class loader that can cause this deadlock.
Suggested patch to solve this is attached.
Here is the deadlock:
Found one Java-level deadlock:
"SLEE-EventRouterExecutor-49-thread-1":
waiting to lock monitor 0x00007f7b0436ca48 (object 0x00000000e8155878, a org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl),
which is held by "SLEE-EventRouterExecutor-39-thread-1"
"SLEE-EventRouterExecutor-39-thread-1":
waiting for ownable synchronizer 0x0000000088a13a28, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "SLEE-EventRouterExecutor-98-thread-1"
"SLEE-EventRouterExecutor-98-thread-1":
waiting to lock monitor 0x00007f7b0436ca48 (object 0x00000000e8155878, a org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl),
which is held by "SLEE-EventRouterExecutor-39-thread-1"
Java stack information for the threads listed above:
"SLEE-EventRouterExecutor-49-thread-1":
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:92)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:55)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.mobicents.slee.runtime.sbb.SbbObjectImpl.createConcreteClass(SbbObjectImpl.java:547)
at org.mobicents.slee.runtime.sbb.SbbObjectImpl.(SbbObjectImpl.java:106)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolFactory.makeObject(SbbObjectPoolFactory.java:146)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolImpl.borrowObject(SbbObjectPoolImpl.java:68)
at org.mobicents.slee.runtime.sbbentity.SbbEntityImpl.assignSbbObject(SbbEntityImpl.java:748)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.routeQueuedEvent(EventRoutingTaskImpl.java:362)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.run(EventRoutingTaskImpl.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
"SLEE-EventRouterExecutor-39-thread-1":
at sun.misc.Unsafe.park(Native Method)
parking to wait for <0x0000000088a13a28> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:442)
at org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl.acquireGlobalLock(URLClassLoaderDomainImpl.java:133)
at org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl.loadClass(URLClassLoaderDomainImpl.java:178)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:92)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:55)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.mobicents.slee.runtime.sbb.SbbObjectImpl.createConcreteClass(SbbObjectImpl.java:547)
at org.mobicents.slee.runtime.sbb.SbbObjectImpl.(SbbObjectImpl.java:106)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolFactory.makeObject(SbbObjectPoolFactory.java:146)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolImpl.borrowObject(SbbObjectPoolImpl.java:68)
at org.mobicents.slee.runtime.sbbentity.SbbEntityImpl.assignSbbObject(SbbEntityImpl.java:748)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.routeQueuedEvent(EventRoutingTaskImpl.java:362)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.run(EventRoutingTaskImpl.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
"SLEE-EventRouterExecutor-98-thread-1":
at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
waiting to lock <0x00000000e8155878> (a org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl)
at org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl.loadClass(URLClassLoaderDomainImpl.java:218)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
locked <0x00000000e8120740> (a org.mobicents.slee.container.component.deployment.classloading.ComponentClassLoaderImpl)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.jboss.as.naming.context.ObjectFactoryBuilder.factoryFromReference(ObjectFactoryBuilder.java:169)
at org.jboss.as.naming.context.ObjectFactoryBuilder.factoryFromReference(ObjectFactoryBuilder.java:158)
at org.jboss.as.naming.context.ObjectFactoryBuilder.createObjectFactory(ObjectFactoryBuilder.java:72)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:299)
at org.restcomm.slee.container.build.as7.naming.RootContext.lookup(RootContext.java:50)
at org.restcomm.slee.container.build.as7.naming.ChildContext.lookup(ChildContext.java:43)
at com.smilecoms.hss.sbb.HSSSbb.setSbbContext(HSSSbb.java:95)
at org.mobicents.slee.runtime.sbb.SbbObjectImpl.(SbbObjectImpl.java:127)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolFactory.makeObject(SbbObjectPoolFactory.java:146)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolImpl.borrowObject(SbbObjectPoolImpl.java:68)
at org.mobicents.slee.runtime.sbbentity.SbbEntityImpl.assignSbbObject(SbbEntityImpl.java:748)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.routeQueuedEvent(EventRoutingTaskImpl.java:362)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.run(EventRoutingTaskImpl.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
We've been experience a deadlock in classloading when using Wildfly. Full details of deadlock below.
It seems in org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl There is a mix of synchronize, global locking and also locking within the default Java class loader that can cause this deadlock.
Suggested patch to solve this is attached.
Here is the deadlock: Found one Java-level deadlock:
"SLEE-EventRouterExecutor-49-thread-1": waiting to lock monitor 0x00007f7b0436ca48 (object 0x00000000e8155878, a org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl), which is held by "SLEE-EventRouterExecutor-39-thread-1" "SLEE-EventRouterExecutor-39-thread-1": waiting for ownable synchronizer 0x0000000088a13a28, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by "SLEE-EventRouterExecutor-98-thread-1" "SLEE-EventRouterExecutor-98-thread-1": waiting to lock monitor 0x00007f7b0436ca48 (object 0x00000000e8155878, a org.mobicents.slee.container.component.deployment.classloading.URLClassLoaderDomainImpl), which is held by "SLEE-EventRouterExecutor-39-thread-1"
Java stack information for the threads listed above:
"SLEE-EventRouterExecutor-49-thread-1": at sun.misc.Unsafe.defineClass(Native Method) at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393) at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:92) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:55) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at org.mobicents.slee.runtime.sbb.SbbObjectImpl.createConcreteClass(SbbObjectImpl.java:547) at org.mobicents.slee.runtime.sbb.SbbObjectImpl.(SbbObjectImpl.java:106)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolFactory.makeObject(SbbObjectPoolFactory.java:146)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.mobicents.slee.runtime.sbb.SbbObjectPoolImpl.borrowObject(SbbObjectPoolImpl.java:68)
at org.mobicents.slee.runtime.sbbentity.SbbEntityImpl.assignSbbObject(SbbEntityImpl.java:748)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.routeQueuedEvent(EventRoutingTaskImpl.java:362)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.run(EventRoutingTaskImpl.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
"SLEE-EventRouterExecutor-39-thread-1":
at sun.misc.Unsafe.park(Native Method)
Found 1 deadlock.