Open jaimecasero opened 7 years ago
particularly in HA configuration, there seems to be issues with closing sockets, and the IOException is fired because the lock for the socket is kept by another thread. This exception is not handled in the stop scenario https://github.com/RestComm/jain-sip/blob/master/src/gov/nist/javax/sip/stack/NIOHandler.java#L508 because we have a silent catch(Exception).
consider to catch Runtime as well and provide at least a WARNING log
Hi jaime ..... i just run into this problem since last week and it's giving me hard time to fix it. I am running Restcomm-Connect-8.6.0-27 and after making a huge quantity of calls I am faced with this issues as well........................ 12:57:25,741 DEBUG [org.mobicents.servlet.sip.core.session.SipApplicationSessionImpl] (Restcomm-SIP-Servlets-UDPMessageChannelThread-61) Before Semaphore released for sipApplicationSession=22406a00;RestComm semaphore=java.util.concurrent.Semaphore@5e342707[Permits = 0] 12:57:25,741 DEBUG [org.mobicents.servlet.sip.core.session.SipApplicationSessionImpl] (Restcomm-SIP-Servlets-UDPMessageChannelThread-61) After Semaphore released for sipApplicationSession=22406a00;RestComm semaphore=java.util.concurrent.Semaphore@5e342707[Permits = 1] 12:57:25,880 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (http-/192.168.153.174:8080-7) checkIsManagedThread false , isManagedThread null, isContainerManaged true 12:57:25,881 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (http-/192.168.153.174:8080-7) NOT RELEASING SipApplicationSession on exit sipApplicationSession=null sipSession=null semaphore=null............................................. (This issue is keeping the tcp connection on my RC server and is not being closed........ is there a way to force kill tcp connection from RC)
10:51:29,433 DEBUG [org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl] (pool-sip_servlets_congestion_control-thread-2) Outbound Interface : sip:10.0.0.28:5082;transport=ws 10:51:29,433 DEBUG [org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl] (pool-sip_servlets_congestion_control-thread-2) Outbound Interface : sip:10.0.0.28:5080;transport=udp 10:51:29,433 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).Executing action:org.mobicents.servlet.sip.core.DispatcherFSM$NotififyConnectorRemoved@3b66d6b1 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 5 container to notify of SIP_CONNECTOR_REMOVED 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) checkIsManagedThread false , isManagedThread null, isContainerManaged true 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container RestComm, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 0 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container SmsService, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 1 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,434 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container CallManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 2 10:51:29,435 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,435 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container UserAgentManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 3 10:51:29,435 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,435 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container ProxyManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 4 10:51:29,435 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,435 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) NOT RELEASING SipApplicationSession on exit sipApplicationSession=null sipSession=null semaphore=null 10:51:29,435 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).New state:IN_SERVICE 10:51:29,436 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).Executing action:org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl$RemovePointAction@442e4b9f 10:51:29,436 DEBUG [org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl] (pool-sip_servlets_congestion_control-thread-2) Outbound Interface List : 10:51:29,436 DEBUG [org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl] (pool-sip_servlets_congestion_control-thread-2) Outbound Interface : sip:10.0.0.28:5080;transport=udp 10:51:29,436 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).Executing action:org.mobicents.servlet.sip.core.DispatcherFSM$NotififyConnectorRemoved@3b66d6b1 10:51:29,436 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 5 container to notify of SIP_CONNECTOR_REMOVED 10:51:29,436 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) checkIsManagedThread false , isManagedThread null, isContainerManaged true 10:51:29,436 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container RestComm, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 0 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container SmsService, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 1 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container CallManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 2 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container UserAgentManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 3 10:51:29,437 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,438 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container ProxyManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 4 10:51:29,438 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,438 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) NOT RELEASING SipApplicationSession on exit sipApplicationSession=null sipSession=null semaphore=null 10:51:29,438 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).New state:IN_SERVICE 10:51:29,438 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).Executing action:org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl$RemovePointAction@442e4b9f 10:51:29,438 DEBUG [org.mobicents.servlet.sip.core.SipNetworkInterfaceManagerImpl] (pool-sip_servlets_congestion_control-thread-2) Outbound Interface List : 10:51:29,438 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).Executing action:org.mobicents.servlet.sip.core.DispatcherFSM$NotififyConnectorRemoved@3b66d6b1 10:51:29,438 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 5 container to notify of SIP_CONNECTOR_REMOVED 10:51:29,438 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) checkIsManagedThread false , isManagedThread null, isContainerManaged true 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container RestComm, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 0 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container SmsService, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 1 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container CallManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 2 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,439 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container UserAgentManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 3 10:51:29,440 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,440 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) container ProxyManager, class : org.mobicents.servlet.sip.catalina.SipServletImpl, load-on-startup : 4 10:51:29,440 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) 0 SipConnectorListener to notify of sip connector removal 10:51:29,440 DEBUG [org.mobicents.servlet.sip.startup.SipStandardContext] (pool-sip_servlets_congestion_control-thread-2) NOT RELEASING SipApplicationSession on exit sipApplicationSession=null sipSession=null semaphore=null 10:51:29,440 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).New state:IN_SERVICE 10:51:29,440 DEBUG [org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl] (pool-sip_servlets_congestion_control-thread-2) State(IN_SERVICE).Executing action:org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl$StopAction@5c991cee 10:51:29,441 WARN [org.mobicents.ha.balancing.only.javax.sip.SipStackImpl] (pool-sip_servlets_congestion_control-thread-2) Could not deregister the stack as a Notification Listener of jboss.system:service=Logging,type=Log4jService runtime changes to log4j.xml won't affect SIP Stack Logging 10:51:29,441 DEBUG [gov.nist.javax.sip.SipStackImpl] (pool-sip_servlets_congestion_control-thread-2) stopStack – stoppping the stack 10:51:29,441 DEBUG [gov.nist.javax.sip.SipStackImpl] (pool-sip_servlets_congestion_control-thread-2) [CommonLoggerLog4j.java:59][CommonLogger.java:116][SipStackImpl.java:1833][SipStackImpl.java:126][ClusteredSipStackImpl.java:192][SipStackImpl.java:78][SipApplicationDispatcherImpl.java:2696][SipApplicationDispatcherImpl.java:568][DispatcherFSM.java:392][DispatcherFSM.java:419][SipApplicationDispatcherImpl.java:588][SipStandardService.java:654][ServiceGracefulStopTask.java:62][Executors.java:471][FutureTask.java:304][ScheduledThreadPoolExecutor.java:178][ScheduledThreadPoolExecutor.java:293][ThreadPoolExecutor.java:1145][ThreadPoolExecutor.java:615][Thread.java:745] 10:51:29,442 DEBUG [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-WSS-10.0.0.28/5083) null selectedKeys 10:51:29,442 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) keys to check for inactivity removal [] 10:51:29,442 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) existing socket in NIOHandler [] 10:51:29,442 INFO [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-WSS-10.0.0.28/5083) Selector is closed 10:51:29,443 DEBUG [gov.nist.javax.sip.stack.UDPMessageProcessor] (MessageProcessorThread-UDP-10.0.0.28/5080) UDPMessageProcessor: Stopping 10:51:29,444 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) keys to check for inactivity removal [] 10:51:29,444 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) existing socket in NIOHandler [] 10:51:29,444 DEBUG [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-WS-10.0.0.28/5082) null selectedKeys 10:51:29,444 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) keys to check for inactivity removal [] 10:51:29,444 INFO [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-WS-10.0.0.28/5082) Selector is closed 10:51:29,444 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) existing socket in NIOHandler [] 10:51:29,445 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) keys to check for inactivity removal [java.nio.channels.SocketChannel[connected local=/10.0.0.28:5080 remote=/10.0.0.29:41463]] 10:51:29,445 DEBUG [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-TCP-10.0.0.28/5080) null selectedKeys 10:51:29,445 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) existing socket in NIOHandler [10.0.0.29:41463] 10:51:29,444 DEBUG [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-TLS-10.0.0.28/5081) null selectedKeys 10:51:29,445 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) stop() : Removing socket tcp:10.0.0.29:41463 socketChannel = java.nio.channels.SocketChannel[connected local=/10.0.0.28:5080 remote=/10.0.0.29:41463] 10:51:29,445 INFO [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-TCP-10.0.0.28/5080) Selector is closed 10:51:29,446 INFO [gov.nist.javax.sip.stack.NioTcpMessageProcessor] (NioSelector-TLS-10.0.0.28/5081) Selector is closed 10:51:29,445 DEBUG [gov.nist.javax.sip.stack.NioTcpMessageChannel] (pool-sip_servlets_congestion_control-thread-2) Closing NioTcpMessageChannel gov.nist.javax.sip.stack.NioTcpMessageChannel@2e789a35 socketChannel = java.nio.channels.SocketChannel[connected local=/10.0.0.28:5080 remote=/10.0.0.29:41463] 10:51:29,446 DEBUG [gov.nist.javax.sip.stack.NioTcpMessageChannel] (pool-sip_servlets_congestion_control-thread-2) Removing NioTcpMessageChannel gov.nist.javax.sip.stack.NioTcpMessageChannel@2e789a35 socketChannel = java.nio.channels.SocketChannel[closed] 10:51:29,447 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) Trying to remove cached socketChannel without keygov.nist.javax.sip.stack.NIOHandler@432caec7 socketChannel = java.nio.channels.SocketChannel[closed] 10:51:29,447 DEBUG [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) Removing cached socketChannel without keygov.nist.javax.sip.stack.NIOHandler@432caec7 socketChannel = java.nio.channels.SocketChannel[closed] key = 10.0.0.29:41463 10:51:29,447 ERROR [gov.nist.javax.sip.stack.NIOHandler] (pool-sip_servlets_congestion_control-thread-2) Failed on putting socket: java.io.IOException: exception in acquiring sem at gov.nist.javax.sip.stack.KeyedSemaphore.enterIOCriticalSection(KeyedSemaphore.java:100) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.NIOHandler.removeSocket(NIOHandler.java:164) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.NIOHandler.removeSocket(NIOHandler.java:224) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.NioTcpMessageChannel.close(NioTcpMessageChannel.java:223) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.ConnectionOrientedMessageChannel.close(ConnectionOrientedMessageChannel.java:127) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.NIOHandler.stop(NIOHandler.java:502) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.NioTcpMessageProcessor.stop(NioTcpMessageProcessor.java:618) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.SIPTransactionStack.removeMessageProcessor(SIPTransactionStack.java:2478) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.stack.SIPTransactionStack.stopStack(SIPTransactionStack.java:2166) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at gov.nist.javax.sip.SipStackImpl.stop(SipStackImpl.java:1835) [jain-sip-ri-7.0.6.315.jar:7.0.6.315] at org.mobicents.ext.javax.sip.SipStackImpl.stop(SipStackImpl.java:126) [mobicents-jain-sip-ext-7.0.4.21.jar:7.0.4.21] at org.mobicents.ha.javax.sip.ClusteredSipStackImpl.stop(ClusteredSipStackImpl.java:192) [mobicents-jain-sip-ha-core-7.0.3.134.jar:7.0.3.134] at org.mobicents.ha.balancing.only.javax.sip.SipStackImpl.stop(SipStackImpl.java:78) [mobicents-jain-sip-ha-core-7.0.3.134.jar:7.0.3.134] at org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl.stopSipStack(SipApplicationDispatcherImpl.java:2696) [sip-servlets-impl-7.0.5.494.jar:7.0.5.494] at org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl$StopAction.execute(SipApplicationDispatcherImpl.java:568) [sip-servlets-impl-7.0.5.494.jar:7.0.5.494] at org.mobicents.servlet.sip.core.DispatcherFSM.executeActions(DispatcherFSM.java:392) [sip-servlets-impl-7.0.5.494.jar:7.0.5.494] at org.mobicents.servlet.sip.core.DispatcherFSM.fireEvent(DispatcherFSM.java:419) [sip-servlets-impl-7.0.5.494.jar:7.0.5.494] at org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl.stop(SipApplicationDispatcherImpl.java:588) [sip-servlets-impl-7.0.5.494.jar:7.0.5.494] at org.mobicents.servlet.sip.catalina.SipStandardService.stop(SipStandardService.java:654) [sip-servlets-as7-7.0.5.494.jar:7.0.5.494] at org.mobicents.servlet.sip.catalina.ServiceGracefulStopTask.run(ServiceGracefulStopTask.java:62) [sip-servlets-as7-7.0.5.494.jar:7.0.5.494] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_95] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [rt.jar:1.7.0_95] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_95] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.7.0_95] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]