SingleEndpointPool race condition

[arinban]

This is the same issue as https://java.net/jira/browse/GRIZZLY-1820. We experienced this hang on a frequent basis, and upgraded the Grizzly libraries to 2.3.24 to pick up the fix for #1820. The upgrade has definitely improved the situation, but we have a Mule app that runs with 8 threads and performs a large number of HTTP requests using a combination of Jersey JAX-RS clients and CXF JAX-WS clients, and the issue still appears.

The symptoms are that some worker threads hang while performing HTTP requests, and CPU usage by the JVM increases to run a single core at 100%.

I will attach the full JVM stack dump, but in essence there is one thread in an infinite loop here:

org.glassfish.grizzly.connectionpool.SingleEndpointPool.cleanupIdleConnections(SingleEndpointPool.java:1106)

And other threads are blocked here:

org.glassfish.grizzly.connectionpool.SingleEndpointPool$PoolConnectionCloseListener.onClosed(SingleEndpointPool.java:1329)

Environment

RHEL 6.6

java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

Mule Enterprise Edition 3.7.1 with Grizzly libraries patched from 2.3.21 to 2.3.24.

Affected Versions

[2.3.24]

[arinban]

• Issue Imported From: https://github.com/javaee/grizzly/issues/1876
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To:@rlubke
• Original Issue Closed By:@glassfishrobot

[arinban]

@glassfishrobot Commented Reported by Mark.Sim-Smith

[arinban]

@glassfishrobot Commented mark.sim-smith said:

Comment too long. Imported partially

Can't attach, so full stack dump here:

Basic info: Date taken: Mon Nov 28 11:31:29 NZDT 2016 File: C:\working\P0_INT14_11_20161128_full_heap_dump.bin File size: 124.6 MB

Total bytes: 103,138,052 Total classes: 20,821 Total instances: 1,277,099 Classloaders: 347 GC roots: 4,711 Number of objects pending for finalization: 0

Environment: OS: Linux (2.6.32-504.23.4.el6.x86_64) Architecture: amd64 64bit Java Home: /usr/lib/jvm/java-1.8.0-oracle-1.8.0.45.x86_64/jre Java Version: 1.8.0_45 JVM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02, mixed mode) Java Vendor: Oracle Corporation

System properties: Show System Properties

Threads at the heap dump:

"hz.595105824.wait-notify" prio=5 tid=19 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#530 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#61 at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) Local Variable: java.util.concurrent.DelayQueue#1 Local Variable: java.util.concurrent.locks.ReentrantLock#75 at com.hazelcast.spi.impl.WaitNotifyServiceImpl$1.run(WaitNotifyServiceImpl.java:66) Local Variable: com.hazelcast.spi.impl.WaitNotifyServiceImpl$1#1 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) Local Variable: java.util.concurrent.Executors$RunnableAdapter#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: java.util.concurrent.FutureTask#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#9 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#4 at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.SingleExecutorThreadFactory$ManagedThread.run(SingleExecutorThreadFactory.java:45)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config.worker(4)" daemon prio=5 tid=131 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#40 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#37 at java.lang.Thread.run(Thread.java:745)

"Wrapper-Connection" daemon prio=10 tid=13 RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) Local Variable: java.io.FileDescriptor#79 Local Variable: byte[]#6436 at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.net.SocketInputStream.read(SocketInputStream.java:223) Local Variable: java.net.SocketInputStream#1 at java.io.DataInputStream.readByte(DataInputStream.java:265) at org.tanukisoftware.wrapper.WrapperManager.handleBackend(WrapperManager.java:5373) Local Variable: org.tanukisoftware.wrapper.event.WrapperPingEvent#1 Local Variable: java.io.DataInputStream#1 at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:5752) Local Variable: org.tanukisoftware.wrapper.WrapperManager#1 at java.lang.Thread.run(Thread.java:745)

"[IFESendInflightData-0-0-1-build-52].http.requester.HTTPRequestConfiguration(3) SelectorRunner" daemon prio=5 tid=177 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#28 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.Util$2#28 Local Variable: java.util.Collections$UnmodifiableSet#148 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#26 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#15 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#42 at java.lang.Thread.run(Thread.java:745)

"[ParkingCustomerSubscriber-0-0-1-build-43].http.requester.HTTPRequestConfiguration.worker(1)" daemon prio=5 tid=111 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#28 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#27 Local Variable: java.util.concurrent.LinkedTransferQueue#4 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#14 at java.lang.Thread.run(Thread.java:745)

"[ParkingCustomerSubscriber-0-0-1-build-43]..endpoint.reconnection" prio=5 tid=120 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#236 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#647 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Local Variable: java.util.concurrent.locks.ReentrantLock#2004 Local Variable: java.util.concurrent.atomic.AtomicInteger#463 Local Variable: java.util.concurrent.LinkedBlockingQueue#37 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor#32 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#21 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.IO.thread-out-2" prio=5 tid=28 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#6 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.EPollSelectorImpl#6 Local Variable: sun.nio.ch.Util$2#6 Local Variable: java.util.Collections$UnmodifiableSet#105 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at com.hazelcast.nio.AbstractIOSelector.run(AbstractIOSelector.java:110)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration(3) SelectorRunner" daemon prio=5 tid=92 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#11 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#121 Local Variable: sun.nio.ch.Util$2#11 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#11 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#1 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#7 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.cached.thread-4" prio=5 tid=45154 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#83 at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#35 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)

"Grizzly-IdleTimeoutFilter-IdleCheck" daemon prio=5 tid=134 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: java.util.concurrent.ThreadPoolExecutor#49 Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#28 at java.lang.Thread.run(Thread.java:745)

"HashSessionScavenger-0" daemon prio=5 tid=52 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: org.eclipse.jetty.server.session.HashSessionManager$2#1 at java.util.TimerThread.run(Timer.java:505)

"RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@812603360[qmid=MQP1.C995248CFB3C7607,fap=11,channel=CH.MQP1.TO.MLETEPP,ccsid=500,sharecnv=10,hbint=300,peer=MQP1.AIRNZ.CO.NZ/10.1.50.189(1721),localport=54528,ssl=no,hConns=0,LastDataSend=1479374245154 (5ms ago),LastDataRecv=1479374245140 (19ms ago),]" daemon prio=5 tid=102 RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) Local Variable: java.io.FileDescriptor#119 Local Variable: java.net.SocketInputStream#3 at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.receive(RemoteTCPConnection.java:1545) Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection#1 Local Variable: byte[]#22267 at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.receiveBuffer(RemoteRcvThread.java:794) at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.receiveOneTSH(RemoteRcvThread.java:757) Local Variable: com.ibm.mq.jmqi.remote.rfp.RfpTSH#13 at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run(RemoteRcvThread.java:150) Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTls#2 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteRcvThread#1 at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:214) at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:105) at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:229) at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:303) Local Variable: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem#1 at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1219)

"HashSessionScavenger-1" daemon prio=5 tid=162 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: org.eclipse.jetty.server.session.HashSessionManager$2#2 at java.util.TimerThread.run(Timer.java:505)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config.worker(5)" daemon prio=5 tid=132 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#39 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#30 at java.lang.Thread.run(Thread.java:745)

"DestroyJavaVM" prio=5 tid=14 RUNNABLE

"[IFESendInflightData-0-0-1-build-52].http.requester.HTTPRequestConfiguration.worker(4)" daemon prio=5 tid=181 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#44 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#47 at java.lang.Thread.run(Thread.java:745)

"Wrapper-Control-Event-Monitor" daemon prio=5 tid=11 TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.tanukisoftware.wrapper.WrapperManager$3.run(WrapperManager.java:1035) Local Variable: org.tanukisoftware.wrapper.WrapperManager$WrapperTickEventImpl#1

"RMI TCP Accept-0" daemon prio=5 tid=164 RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) Local Variable: java.net.SocksSocketImpl#7 at java.net.ServerSocket.implAccept(ServerSocket.java:545) Local Variable: java.net.SocksSocketImpl#20 at java.net.ServerSocket.accept(ServerSocket.java:513) Local Variable: java.net.Socket#15 at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372) Local Variable: sun.rmi.transport.tcp.TCPTransport$AcceptLoop#1 at java.lang.Thread.run(Thread.java:745)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration.worker(1)" daemon prio=5 tid=93 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue#1 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#4 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#3 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#6 at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-51" prio=5 tid=51 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#610 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-3" prio=5 tid=61 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"hz.595105824.cached.thread-2" prio=5 tid=37792 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#77 at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#34 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)

"hz.595105824.IO.thread-in-1" prio=5 tid=25 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#3 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.EPollSelectorImpl#3 Local Variable: java.util.Collections$UnmodifiableSet#102 Local Variable: sun.nio.ch.Util$2#3 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at com.hazelcast.nio.AbstractIOSelector.run(AbstractIOSelector.java:110)

"[ParkingCustomerSubscriber-0-0-1-build-43].http.requester.HTTPRequestConfiguration.worker(5)" daemon prio=5 tid=115 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#25 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#13 at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-50" prio=5 tid=50 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#611 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"asyncDelivery0" daemon prio=5 tid=185 RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) Local Variable: java.net.SocketInputStream#6 Local Variable: java.io.FileDescriptor#155 at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.receive(RemoteTCPConnection.java:1545) Local Variable: byte[]#23306 at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveBuffer(RemoteSession.java:917) at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveOneTSH(RemoteSession.java:882) Local Variable: com.ibm.mq.jmqi.remote.rfp.RfpTSH#2 at com.ibm.mq.jmqi.remote.impl.RemoteSession.receive1stGetReplyTSH(RemoteSession.java:1463) at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetMessageWithProps(RemoteFAP.java:8577) Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection#1 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteSession#1 Local Variable: com.ibm.mq.jmqi.internal.JmqiDC#1 Local Variable: com.ibm.mq.jmqi.system.JmqiCodepage#2 Local Variable: com.ibm.mq.jmqi.system.JmqiTls#1 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTls#1 at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQGET(RemoteFAP.java:8341) Local Variable: com.ibm.mq.jmqi.remote.api.RemoteHobj#2 Local Variable: com.ibm.mq.jmqi.remote.api.RemoteFAP#1 Local Variable: com.ibm.mq.jmqi.MQMD#1 at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.MQGET(InterceptedJmqiImpl.java:815) at com.ibm.mq.ese.jmqi.ESEJMQI.MQGET(ESEJMQI.java:258) at com.ibm.msg.client.wmq.v6.base.internal.MQSESSION.MQGET(MQSESSION.java:984) Local Variable: com.ibm.msg.client.wmq.v6.base.internal.MQSESSION#1 Local Variable: com.ibm.mq.jmqi.remote.api.RemoteHconn#1 Local Variable: com.ibm.mq.jmqi.MQGMO#1 at com.ibm.msg.client.wmq.v6.base.internal.MQQueue.getMsg2Int(MQQueue.java:1315) Local Variable: java.nio.HeapByteBuffer#41 Local Variable: com.ibm.mq.jmqi.handles.Pint#2 Local Variable: byte[]#9747 Local Variable: byte[]#9750 Local Variable: byte[]#9749 Local Variable: byte[]#9748 Local Variable: com.ibm.mq.jmqi.handles.Pint#16 Local Variable: com.ibm.mq.jmqi.handles.Pint#15 Local Variable: com.ibm.msg.client.wmq.v6.jms.internal.MQJMSMessage#1 Local Variable: com.ibm.msg.client.wmq.v6.base.internal.MQGetMessageOptions#2 at com.ibm.msg.client.wmq.v6.base.internal.MQQueue.getMsg2(MQQueue.java:959) Local Variable: com.ibm.msg.client.wmq.v6.base.internal.MQQueue#1 at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.getMessage(MQMessageConsumer.java:2705) at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.receiveAsync(MQMessageConsumer.java:4183) at com.ibm.msg.client.wmq.v6.jms.internal.SessionAsyncHelper.run(SessionAsyncHelper.java:525) Local Variable: com.ibm.msg.client.wmq.v6.jms.internal.SessionAsyncHelper#1 Local Variable: com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer#1 at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-2" prio=5 tid=60 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration.worker(2)" daemon prio=5 tid=94 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#1 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#5 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#5 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config.worker(3)" daemon prio=5 tid=130 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#41 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#31 at java.lang.Thread.run(Thread.java:745)

"JmDNS(akl0mw904-p0-airnz-co-nz.local.).Timer" daemon prio=5 tid=54 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: javax.jmdns.impl.tasks.RecordReaper#2 at java.util.TimerThread.run(Timer.java:505)

"qtp1208852147-48" prio=5 tid=48 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#103 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#613 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) Local Variable: org.eclipse.jetty.util.BlockingArrayQueue#1 Local Variable: java.util.concurrent.locks.ReentrantLock#1176 at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-1" prio=5 tid=59 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"[ParkingCustomerSubscriber-0-0-1-build-43].http.requester.HTTPRequestConfiguration(1) SelectorRunner" daemon prio=5 tid=108 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#14 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#127 Local Variable: sun.nio.ch.Util$2#14 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#14 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) Local Variable: org.glassfish.grizzly.utils.StateHolder#5 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#5 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#18 at java.lang.Thread.run(Thread.java:745)

"Timer-1" daemon prio=5 tid=152 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: java.lang.Object#29686 Local Variable: org.quartz.utils.UpdateChecker#2 at java.util.TimerThread.run(Timer.java:505)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration.worker(3)" daemon prio=5 tid=95 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#4 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.operation.thread-0" prio=5 tid=75 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#32 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#607 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Local Variable: java.util.concurrent.LinkedBlockingQueue#15 Local Variable: java.util.concurrent.locks.ReentrantLock#46 Local Variable: java.util.concurrent.atomic.AtomicInteger#24 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor#2 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#14 at java.lang.Thread.run(Thread.java:745) at com.hazelcast.spi.impl.OperationServiceImpl$OperationThread.run(OperationServiceImpl.java:783)

"[IFESendInflightData-0-0-1-build-52].http.requester.HTTPRequestConfiguration.worker(5)" daemon prio=5 tid=182 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#47 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#48 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.cached.thread-5" prio=5 tid=37253 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) Local Variable: java.util.concurrent.SynchronousQueue$TransferStack#1 Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#89 at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#33 Local Variable: java.util.concurrent.ThreadPoolExecutor#1 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)

"qtp1011158714-154-selector-0" prio=5 tid=154 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#20 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.EPollSelectorImpl#20 Local Variable: sun.nio.ch.Util$2#20 Local Variable: java.util.Collections$UnmodifiableSet#134 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) Local Variable: java.lang.String#115841 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) Local Variable: org.eclipse.jetty.util.thread.QueuedThreadPool#2 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) Local Variable: org.eclipse.jetty.util.thread.QueuedThreadPool$3#2 Local Variable: org.eclipse.jetty.io.SelectorManager$ManagedSelector#4 at java.lang.Thread.run(Thread.java:745)

"[IFESendInflightData-0-0-1-build-52].IFESendInflightData.1.01" prio=5 tid=184 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.mule.util.queue.AbstractQueueStoreDelegate.poll(AbstractQueueStoreDelegate.java:81) at org.mule.util.queue.DefaultQueueStore.poll(DefaultQueueStore.java:89) at org.mule.util.queue.CacheAwareQueueStore.poll(CacheAwareQueueStore.java:48) at org.mule.util.queue.TransactionAwareQueueStore.poll(TransactionAwareQueueStore.java:105) Local Variable: org.mule.util.queue.TransactionAwareQueueStore#1 at org.mule.processor.SedaStageInterceptingMessageProcessor.dequeue(SedaStageInterceptingMessageProcessor.java:155) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:230) Local Variable: com.mulesoft.mule.cluster.hazelcast.vm.ClusterQueueSession#1 Local Variable: org.mule.processor.SedaStageInterceptingMessageProcessor#1 at org.mule.work.WorkerContext.run(WorkerContext.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: com.mulesoft.mule.config.pool.MonitoredThreadPoolExecutor#53 Local Variable: org.mule.work.WorkerContext#22 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#30 at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-49" prio=5 tid=49 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#612 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"SocketListener(akl0mw904-airnz-co-nz.local.)" daemon prio=5 tid=56 RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:144) at java.net.DatagramSocket.receive(DatagramSocket.java:812) Local Variable: java.net.MulticastSocket#1 at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) Local Variable: byte[]#18201 Local Variable: java.net.DatagramPacket#1

"RMI Scheduler(0)" daemon prio=5 tid=167 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#281 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#673 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#24 Local Variable: java.util.concurrent.locks.ReentrantLock#2272 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#24 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#24 at java.lang.Thread.run(Thread.java:745)

"[ParkingCustomerSubscriber-0-0-1-build-43].processing.time.monitor" daemon prio=5 tid=121 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#4124 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76) Local Variable: org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker#4 at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-47-acceptor-0-ServerConnector@379d3eca

{HTTP/1.1}{0.0.0.0:7780}" prio=5 tid=47 RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) Local Variable: java.io.FileDescriptor#91 at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) Local Variable: java.net.InetSocketAddress[]#2 Local Variable: java.io.FileDescriptor#102 Local Variable: java.lang.Object#6102 at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:321) Local Variable: sun.nio.ch.ServerSocketChannelImpl#2 at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:460) Local Variable: java.lang.String#51282 Local Variable: org.eclipse.jetty.server.ServerConnector#1 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) Local Variable: org.eclipse.jetty.server.AbstractConnector$Acceptor#1 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config.worker(6)" daemon prio=5 tid=133 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#38 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#34 at java.lang.Thread.run(Thread.java:745)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration.worker(4)" daemon prio=5 tid=96 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#6 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#51 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#3 at java.lang.Thread.run(Thread.java:745)

"[TicketEventParkingSub-0-0-1-build-45]..endpoint.reconnection" prio=5 tid=103 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#645 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#216 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Local Variable: java.util.concurrent.atomic.AtomicInteger#426 Local Variable: java.util.concurrent.locks.ReentrantLock#1938 Local Variable: java.util.concurrent.LinkedBlockingQueue#36 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor#31 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#19 at java.lang.Thread.run(Thread.java:745)

"scheduler-monitoring-0-0-5_mule3.6-build-1_Worker-1" prio=5 tid=150 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"[IFESendInflightData-0-0-1-build-52].http.requester.HTTPRequestConfiguration.worker(3)" daemon prio=5 tid=180 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#45 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#43 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#46 at java.lang.Thread.run(Thread.java:745)

"[ParkingCustomerSubscriber-0-0-1-build-43].http.requester.HTTPRequestConfiguration.worker(4)" daemon prio=5 tid=114 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#26 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#12 at java.lang.Thread.run(Thread.java:745)

"Grizzly-IdleTimeoutFilter-IdleCheck" daemon prio=5 tid=106 BLOCKED at org.glassfish.grizzly.connectionpool.SingleEndpointPool$PoolConnectionCloseListener.onClosed(SingleEndpointPool.java:1329) Local Variable: org.glassfish.grizzly.connectionpool.SingleEndpointPool$PoolConnectionCloseListener#3 at org.glassfish.grizzly.connectionpool.SingleEndpointPool$PoolConnectionCloseListener.onClosed(SingleEndpointPool.java:1323) at org.glassfish.grizzly.nio.NIOConnection.invokeCloseListener(NIOConnection.java:1001) at org.glassfish.grizzly.nio.NIOConnection.notifyCloseListeners(NIOConnection.java:871) Local Variable: org.glassfish.grizzly.CloseType#1 at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:604) at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) at org.glassfish.grizzly.nio.NIOConnection$4.completed(NIOConnection.java:567) at org.glassfish.grizzly.nio.NIOConnection$4.completed(NIOConnection.java:563) at org.glassfish.grizzly.asyncqueue.AsyncWriteQueueRecord.notifyCompleteAndRecycle(AsyncWriteQueueRecord.java:173) Local Variable: org.glassfish.grizzly.memory.Buffers$1#1 at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:279) Local Variable: org.glassfish.grizzly.asyncqueue.AsyncWriteQueueRecord#11 Local Variable: org.glassfish.grizzly.Writer$Reentrant#11 at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) at org.glassfish.grizzly.AbstractWriter.write(AbstractWriter.java:76) at org.glassfish.grizzly.nio.NIOConnection.closeGracefully0(NIOConnection.java:562) at org.glassfish.grizzly.nio.NIOConnection.closeWithReason(NIOConnection.java:513) at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:431) at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:274) at org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:401) at org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:380) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158) Local Variable: java.util.concurrent.ConcurrentLinkedQueue$Itr#80 Local Variable: java.util.concurrent.ConcurrentHashMap$KeyIterator#94 Local Variable: org.glassfish.grizzly.utils.IdleTimeoutFilter$Resolver#2 Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayQueue#6 Local Variable: org.glassfish.grizzly.nio.transport.TCPNIOConnection#2 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#6 Local Variable: java.util.concurrent.ThreadPoolExecutor#34 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#22 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.IO.thread-Acceptor" prio=5 tid=29 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#7 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#106 Local Variable: sun.nio.ch.Util$2#7 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at com.hazelcast.nio.SocketAcceptor.run(SocketAcceptor.java:47) Local Variable: sun.nio.ch.EPollSelectorImpl#7 Local Variable: com.hazelcast.nio.SocketAcceptor#1 at java.lang.Thread.run(Thread.java:745)

"[.agent].processing.time.monitor" daemon prio=5 tid=41 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#739 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76) Local Variable: org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker#1 at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-7" prio=5 tid=65 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration.worker(5)" daemon prio=5 tid=97 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#2 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.HealthMonitor" daemon prio=5 tid=32 TIMED_WAITING at java.lang.Thread.sleep(Native Method) at com.hazelcast.util.HealthMonitor.run(HealthMonitor.java:106) Local Variable: com.hazelcast.util.HealthMonitor$HealthMetrics#1

"RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@1550948976[qmid=MQP1.C995248CFB3C7607,fap=11,channel=CH.MQP1.TO.MLEPCPP,ccsid=500,sharecnv=10,hbint=300,peer=MQP1.AIRNZ.CO.NZ/10.1.50.189(1721),localport=42439,ssl=no,hConns=0,LastDataSend=1479374250840 (4ms ago),LastDataRecv=1479374250827 (17ms ago),]" daemon prio=5 tid=119 RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) Local Variable: java.net.SocketInputStream#4 Local Variable: java.io.FileDescriptor#126 at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.receive(RemoteTCPConnection.java:1545) Local Variable: byte[]#16111 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection#1 at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.receiveBuffer(RemoteRcvThread.java:794) at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.receiveOneTSH(RemoteRcvThread.java:757) Local Variable: com.ibm.mq.jmqi.remote.rfp.RfpTSH#1 at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run(RemoteRcvThread.java:150) Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTls#2 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteRcvThread#1 at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:214) at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:105) at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:229) at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:303) Local Variable: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem#1 at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1219)

"hz.595105824.scheduled" prio=5 tid=21 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#18 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#554 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) Local Variable: java.util.concurrent.locks.ReentrantLock#27 Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#1 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#1 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#5 at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.SingleExecutorThreadFactory$ManagedThread.run(SingleExecutorThreadFactory.java:45)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].Mule.01" prio=5 tid=122 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#235 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#572 at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) Local Variable: java.util.concurrent.locks.ReentrantLock#2002 Local Variable: java.util.concurrent.LinkedBlockingDeque#5 at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.mule.context.notification.ServerNotificationManager.run(ServerNotificationManager.java:267) Local Variable: org.mule.context.notification.ServerNotificationManager#5 at org.mule.work.WorkerContext.run(WorkerContext.java:286) Local Variable: org.mule.module.launcher.application.CompositeApplicationClassLoader#4 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: org.mule.work.WorkerContext#19 Local Variable: com.mulesoft.mule.config.pool.MonitoredThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#20 at java.lang.Thread.run(Thread.java:745)

"hz.595105824.IO.thread-out-1" prio=5 tid=26 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#4 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.EPollSelectorImpl#4 Local Variable: sun.nio.ch.Util$2#4 Local Variable: java.util.Collections$UnmodifiableSet#103 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at com.hazelcast.nio.AbstractIOSelector.run(AbstractIOSelector.java:110)

"JMSCCThreadPoolMaster" daemon prio=5 tid=81 WAITING at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$WorkQueueManagerThread.waitForNotification(WorkQueueManagerImplementation.java:660) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$WorkQueueManagerThread.waitForNotification(WorkQueueManagerImplementation.java:629) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$WorkQueueManagerThread.run(WorkQueueManagerImplementation.java:897) Local Variable: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem#1

"scheduler_Worker-8" prio=5 tid=66 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"qtp1011158714-161" prio=5 tid=161 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#595 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-46-selector-2" prio=5 tid=46 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#8 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#113 Local Variable: sun.nio.ch.Util$2#8 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) Local Variable: sun.nio.ch.EPollSelectorImpl#8 at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) Local Variable: java.lang.String#51283 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) Local Variable: org.eclipse.jetty.io.SelectorManager$ManagedSelector#1 at java.lang.Thread.run(Thread.java:745)

"[TicketEventParkingSub-0-0-1-build-45].http.requester.HTTP_Request_Configuration.worker(6)" daemon prio=5 tid=98 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#2 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#1 at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-44-selector-0" prio=5 tid=44 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#10 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#115 Local Variable: sun.nio.ch.Util$2#10 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) Local Variable: sun.nio.ch.EPollSelectorImpl#10 at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) Local Variable: java.lang.String#51422 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) Local Variable: org.eclipse.jetty.util.thread.QueuedThreadPool#1 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) Local Variable: org.eclipse.jetty.util.thread.QueuedThreadPool$3#1 Local Variable: org.eclipse.jetty.io.SelectorManager$ManagedSelector#3 at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-4" prio=5 tid=62 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"hz.595105824.operation.thread-5" prio=5 tid=80 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#603 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#47 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Local Variable: java.util.concurrent.locks.ReentrantLock#61 Local Variable: java.util.concurrent.atomic.AtomicInteger#34 Local Variable: java.util.concurrent.LinkedBlockingQueue#20 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor#7 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#10 at java.lang.Thread.run(Thread.java:745) at com.hazelcast.spi.impl.OperationServiceImpl$OperationThread.run(OperationServiceImpl.java:783)

"connection-pool-delays-thread-pool(1)" daemon prio=5 tid=89 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#4 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#10 at java.lang.Thread.run(Thread.java:745)

"qtp1011158714-159" prio=5 tid=159 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#597 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"qtp1208852147-45-selector-1" prio=5 tid=45 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#9 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#114 Local Variable: sun.nio.ch.Util$2#9 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) Local Variable: sun.nio.ch.EPollSelectorImpl#9 at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) Local Variable: java.lang.String#51413 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) Local Variable: org.eclipse.jetty.io.SelectorManager$ManagedSelector#2 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config(1) SelectorRunner" daemon prio=5 tid=125 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#24 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#141 Local Variable: sun.nio.ch.Util$2#24 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#24 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) Local Variable: org.glassfish.grizzly.utils.StateHolder#7 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#11 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#40 at java.lang.Thread.run(Thread.java:745)

"[TicketEventParkingSub-0-0-1-build-45].processing.time.monitor" daemon prio=5 tid=104 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#3821 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76) Local Variable: org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker#3 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_OAuth_Config.worker(4)" daemon prio=5 tid=142 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#29 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#24 at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-5" prio=5 tid=63 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"qtp1011158714-157" prio=5 tid=157 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#288 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#598 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) Local Variable: org.eclipse.jetty.util.BlockingArrayQueue#2 Local Variable: java.util.concurrent.locks.ReentrantLock#2320 at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"[monitoring-0-0-5_mule3.6-build-1].Mule.01" prio=5 tid=149 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#282 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#569 at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) Local Variable: java.util.concurrent.locks.ReentrantLock#2308 Local Variable: java.util.concurrent.LinkedBlockingDeque#7 at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.mule.context.notification.ServerNotificationManager.run(ServerNotificationManager.java:267) Local Variable: org.mule.context.notification.ServerNotificationManager#7 at org.mule.work.WorkerContext.run(WorkerContext.java:286) Local Variable: org.mule.module.launcher.application.CompositeApplicationClassLoader#6 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: com.mulesoft.mule.config.pool.MonitoredThreadPoolExecutor#35 Local Variable: org.mule.work.WorkerContext#21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#25 at java.lang.Thread.run(Thread.java:745)

"JMSCCThreadPoolMaster" daemon prio=5 tid=118 WAITING at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$WorkQueueManagerThread.waitForNotification(WorkQueueManagerImplementation.java:660) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$WorkQueueManagerThread.waitForNotification(WorkQueueManagerImplementation.java:629) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$WorkQueueManagerThread.run(WorkQueueManagerImplementation.java:897) Local Variable: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem#2

"connection-pool-delays-thread-pool(1)" daemon prio=5 tid=174 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#16 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#41 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_OAuth_Config(2) SelectorRunner" daemon prio=5 tid=137 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#17 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.Util$2#17 Local Variable: java.util.Collections$UnmodifiableSet#131 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#17 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#10 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#28 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config(3) SelectorRunner" daemon prio=5 tid=127 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#25 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.Util$2#25 Local Variable: java.util.Collections$UnmodifiableSet#143 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#25 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#12 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#38 at java.lang.Thread.run(Thread.java:745)

"JmDNS(akl0mw904-airnz-co-nz.local.).State.Timer" prio=5 tid=58 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: javax.jmdns.impl.tasks.state.Renewer#1 at java.util.TimerThread.run(Timer.java:505)

"qtp1011158714-160" prio=5 tid=160 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#596 at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:745)

"DateCache" daemon prio=5 tid=43 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: org.eclipse.jetty.server.Server$1#1 at java.util.TimerThread.run(Timer.java:505)

"scheduler_Worker-6" prio=5 tid=64 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"Timer-0" daemon prio=5 tid=70 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: org.quartz.utils.UpdateChecker#1 Local Variable: java.lang.Object#10398 at java.util.TimerThread.run(Timer.java:505)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_OAuth_Config.worker(5)" daemon prio=5 tid=143 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#37 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#22 at java.lang.Thread.run(Thread.java:745)

"[ParkingCustomerSubscriber-0-0-1-build-43].http.requester.HTTPRequestConfiguration.worker(6)" daemon prio=5 tid=116 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#11 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventPub-0-0-1-build-40].processing.time.monitor" daemon prio=5 tid=78 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#1682 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76) Local Variable: org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker#2 at java.lang.Thread.run(Thread.java:745)

"[IFESendInflightData-0-0-1-build-52].http.requester.HTTPRequestConfiguration(1) SelectorRunner" daemon prio=5 tid=175 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#27 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: java.util.Collections$UnmodifiableSet#147 Local Variable: sun.nio.ch.Util$2#27 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#28 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) Local Variable: org.glassfish.grizzly.utils.StateHolder#11 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#13 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#43 at java.lang.Thread.run(Thread.java:745)

"[monitoring-0-0-5_mule3.6-build-1].processing.time.monitor" daemon prio=5 tid=170 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#4381 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76) Local Variable: org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker#6 at java.lang.Thread.run(Thread.java:745)

"scheduler_QuartzSchedulerThread" prio=5 tid=69 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)

"JmDNS(akl0mw904-airnz-co-nz.local.).Timer" daemon prio=5 tid=57 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: javax.jmdns.impl.tasks.RecordReaper#1 at java.util.TimerThread.run(Timer.java:505)

"hz.595105824.operation.thread-2" prio=5 tid=35 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#38 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#615 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Local Variable: java.util.concurrent.LinkedBlockingQueue#17 Local Variable: java.util.concurrent.atomic.AtomicInteger#28 Local Variable: java.util.concurrent.locks.ReentrantLock#52 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) Local Variable: java.util.concurrent.ThreadPoolExecutor#4 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#6 at java.lang.Thread.run(Thread.java:745) at com.hazelcast.spi.impl.OperationServiceImpl$OperationThread.run(OperationServiceImpl.java:783)

"Reference Handler" daemon prio=10 tid=2 WAITING at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config.worker(2)" daemon prio=5 tid=129 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#42 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#36 at java.lang.Thread.run(Thread.java:745)

"qtp1011158714-158-acceptor-0-ServerConnector@7df74c66{HTTP/1.1} {akl0mw904-p0.airnz.co.nz:30053}

" prio=5 tid=158 RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) Local Variable: sun.nio.ch.ServerSocketChannelImpl#3 Local Variable: java.lang.Object#29743 Local Variable: java.net.InetSocketAddress[]#1 Local Variable: java.io.FileDescriptor#13 at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:321) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:460) Local Variable: org.eclipse.jetty.server.ServerConnector#2 Local Variable: java.lang.String#115839 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) Local Variable: org.eclipse.jetty.server.AbstractConnector$Acceptor#2 at java.lang.Thread.run(Thread.java:745)

"connection-pool-delays-thread-pool(1)" daemon prio=5 tid=135 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#12 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#33 at java.lang.Thread.run(Thread.java:745)

"[IFESendInflightData-0-0-1-build-52].processing.time.monitor" daemon prio=5 tid=186 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#4624 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76) Local Variable: org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker#7 at java.lang.Thread.run(Thread.java:745)

"RMI Reaper" prio=5 tid=165 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#4382 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351) Local Variable: sun.rmi.transport.ObjectTable$Reaper#1 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_OAuth_Config.worker(2)" daemon prio=5 tid=140 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) Local Variable: java.util.concurrent.LinkedTransferQueue$Node#31 Local Variable: java.util.concurrent.LinkedTransferQueue$Node#32 at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265) at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:583) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#23 at java.lang.Thread.run(Thread.java:745)

"asyncDelivery0" daemon prio=5 tid=148 RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) Local Variable: java.net.SocketInputStream#5 Local Variable: java.io.FileDescriptor#137 at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.receive(RemoteTCPConnection.java:1545) Local Variable: byte[]#23289 at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveBuffer(RemoteSession.java:917) at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveOneTSH(RemoteSession.java:882) Local Variable: com.ibm.mq.jmqi.remote.rfp.RfpTSH#3 at com.ibm.mq.jmqi.remote.impl.RemoteSession.receive1stGetReplyTSH(RemoteSession.java:1463) at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetMessageWithProps(RemoteFAP.java:8577) Local Variable: com.ibm.mq.jmqi.system.JmqiCodepage#2 Local Variable: com.ibm.mq.jmqi.internal.JmqiDC#1 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteSession#1 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTls#1 Local Variable: com.ibm.mq.jmqi.system.JmqiTls#1 Local Variable: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection#1 at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQGET(RemoteFAP.java:8341) Local Variable: com.ibm.mq.jmqi.MQMD#1 Local Variable: com.ibm.mq.jmqi.remote.api.RemoteHobj#1 Local Variable: com.ibm.mq.jmqi.remote.api.RemoteFAP#1 at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.MQGET(InterceptedJmqiImpl.java:815) at com.ibm.mq.ese.jmqi.ESEJMQI.MQGET(ESEJMQI.java:258) at com.ibm.msg.client.wmq.v6.base.internal.MQSESSION.MQGET(MQSESSION.java:984) Local Variable: com.ibm.mq.jmqi.MQGMO#1 Local Variable: com.ibm.mq.jmqi.remote.api.RemoteHconn#1 Local Variable: com.ibm.msg.client.wmq.v6.base.internal.MQSESSION#1 at com.ibm.msg.client.wmq.v6.base.internal.MQQueue.getMsg2Int(MQQueue.java:1315) Local Variable: com.ibm.mq.jmqi.handles.Pint#8 Local Variable: com.ibm.mq.jmqi.handles.Pint#9 Local Variable: byte[]#16093 Local Variable: byte[]#16092 Local Variable: com.ibm.mq.jmqi.handles.Pint#3 Local Variable: java.nio.HeapByteBuffer#52 Local Variable: byte[]#16095 Local Variable: byte[]#16094 Local Variable: com.ibm.msg.client.wmq.v6.base.internal.MQGetMessageOptions#1 Local Variable: com.ibm.msg.client.wmq.v6.jms.internal.MQJMSMessage#1 at com.ibm.msg.client.wmq.v6.base.internal.MQQueue.getMsg2(MQQueue.java:959) Local Variable: com.ibm.msg.client.wmq.v6.base.internal.MQQueue#1 at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.getMessage(MQMessageConsumer.java:2705) at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.receiveAsync(MQMessageConsumer.java:4183) at com.ibm.msg.client.wmq.v6.jms.internal.SessionAsyncHelper.run(SessionAsyncHelper.java:525) Local Variable: com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer#1 Local Variable: com.ibm.msg.client.wmq.v6.jms.internal.SessionAsyncHelper#1 at java.lang.Thread.run(Thread.java:745)

"Grizzly-IdleTimeoutFilter-IdleCheck" daemon prio=5 tid=88 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#3 Local Variable: java.util.concurrent.ThreadPoolExecutor#25 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#17 at java.lang.Thread.run(Thread.java:745)

"Grizzly-IdleTimeoutFilter-IdleCheck" daemon prio=5 tid=123 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: java.util.concurrent.ThreadPoolExecutor#47 Local Variable: org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable#10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#27 at java.lang.Thread.run(Thread.java:745)

"[EmployeeExitEventNewsAppSub-0-0-1-build-19].http.requester.HTTP_Delete_Config(2) SelectorRunner" daemon prio=5 tid=126 RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) Local Variable: sun.nio.ch.EPollArrayWrapper#23 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) Local Variable: sun.nio.ch.Util$2#23 Local Variable: java.util.Collections$UnmodifiableSet#140 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115) Local Variable: sun.nio.ch.EPollSelectorImpl#23 at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:339) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) Local Variable: org.glassfish.grizzly.nio.SelectorRunner#9 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) Local Variable: org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker#39 at java.lang.Thread.run(Thread.java:745)

"[TicketEventParkingSub-0-0-1-build-45].Mule.01" prio=5 tid=86 TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#172 Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#581 at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) Local Variable: java.util.concurrent.locks.ReentrantLock#1530 Local Variable: java.util.concurrent.LinkedBlockingDeque#3 at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.mule.context.notification.ServerNotificationManager.run(ServerNotificationManager.java:267) Local Variable: org.mule.context.notification.ServerNotificationManager#3 at org.mule.work.WorkerContext.run(WorkerContext.java:286) Local Variable: org.mule.module.launcher.application.CompositeApplicationClassLoader#2 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Local Variable: org.mule.work.WorkerContext#17 Local Variable: com.mulesoft.mule.config.pool.MonitoredThreadPoolExecutor#17 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#16 at java.lang.Thread.run(Thread.java:745)

"scheduler_Worker-9" prio=5 tid=67 TIMED_WAITING at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

"Finalizer" daemon prio=8 tid=3 WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) Local Variable: java.lang.ref.ReferenceQueue#167 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Local Variable: java.lang.System$2#1

"JmDNS(akl0mw904-p0-airnz-co-nz.local.).State.Timer" prio=5 tid=55 TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) Local Variable: javax.jmdns.impl.tasks.state.Renewer#3 at java.util.TimerThread.run(Timer.java:505)

"DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=C3E2D8C3D4D8D7F14040404040404040D1A8BF175BAD0001]]" daemon prio=5 tid=188 BLOCKED at org.glassfish.grizzly.connectionpool.SingleEndpointPool.take(SingleEndpointPool.java:755) at org.glassfish.grizzly.connectionpool.MultiEndpointPool.take(MultiEndpointPool.java:592) Local Variable: com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1#2 at com.ning.http.client.providers.grizzly.ConnectionManager.openAsync(ConnectionManager.java:142) at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:174) Local Variable: com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#2 Local Variable: com.ning.http.client.providers.grizzly.GrizzlyResponseFuture#6 at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:506) at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:528) at org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient.send(GrizzlyHttpClient.java:204) Local Variable: org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient#2 at org.mule.module.http.internal.request.DefaultHttpRequester.innerProcess(DefaultHttpRequester.java:282) Local Variable: org.mule.module.http.internal.request.DefaultHttpRequester#5 Local Variable: org.mule.module.http.internal.domain.request.DefaultHttpRequest#2 at org.mule.module.http.internal.request.DefaultHttpRequester.processBlocking(DefaultHttpRequester.java:186) at org.mule.processor.AbstractNonBlockingMessageProcessor.process(AbstractNonBlockingMessageProcessor.java:43) at org.mule.module.ws.consumer.WSConsumerConfig$1.process(WSConsumerConfig.java:127) at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24) at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107) Local Variable: org.mule.module.ws.consumer.WSConsumerConfig$1#6 Local Variable: org.mule.execution.MessageProcessorNotificationExecutionInterceptor#194 at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44) at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:94) at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:56) Local Variable: org.mule.processor.BlockingProcessorExecutor#62 at org.mule.processor.AbstractInterceptingMessageProcessorBase.processNext(AbstractInterceptingMessageProcessorBase.java:98) at org.mule.processor.AbstractRequestResponseMessageProcessor.processBlocking(AbstractRequestResponseMessageProcessor.java:56) at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:47) at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24) at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107) Local Variable: org.mule.execution.MessageProcessorNotificationExecutionInterceptor#195 Local Variable: org.mule.module.ws.consumer.WSConsumer$3#6 at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44) at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:94) at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:56) Local Variable: org.mule.processor.BlockingProcessorExecutor#63 at org.mule.processor.AbstractInterceptingMessageProcessorBase.processNext(AbstractInterceptingMessageProcessorBase.java:98) at org.mule.module.cxf.CxfOutboundMessageProcessor.processNext(CxfOutboundMessageProcessor.java:179) at org.mule.module.cxf.transport.MuleUniversalConduit.processNext(MuleUniversalConduit.java:387) at org.mule.module.cxf.transport.MuleUniversalConduit.dispatchMuleMessage(MuleUniversalConduit.java:282) Local Variable: org.mule.module.cxf.transport.MuleUniversalConduit#6 at org.mule.module.cxf.transport.MuleUniversalConduit$2.handleMessage(MuleUniversalConduit.java:193) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) Local Variable: org.apache.cxf.phase.PhaseInterceptorChain#17 at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:572) Local Variable: org.apache.cxf.service.model.BindingOperationInfo#12 Local Variable: org.apache.cxf.endpoint.ClientImpl#6 Local Variable: org.apache.cxf.message.ExchangeImpl#2 Local Variable: org.apache.cxf.binding.soap.SoapMessage#3 Local Variable: java.util.HashMap#16244 at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:481) at org.mule.module.cxf.CxfOutboundM

[arinban]

@glassfishrobot Commented @rlubke said: Thanks for the report. Would you mind trying this jar [1] to see if it resolves your issue?

[1] https://www.dropbox.com/s/bdfva1l0fdj9l5b/connection-pool-2.3.29-SNAPSHOT.jar?dl=0

[arinban]

@glassfishrobot Commented mark.sim-smith said: Hi Ryan,

I will give it a go, although it may be a couple of weeks before we will know.

Cheers,

Mark

[arinban]

@glassfishrobot Commented @rlubke said: Hey Mark,

Any update?

[arinban]

@glassfishrobot Commented mark.sim-smith said: Hi Ryan, sorry for the delay in responding. After a bit of analysis we realised that we had upgraded to Grizzly version 2.3.24 for all grizzly libraries, except connection-pool (which was missed because it wasn't prefixed with "grizzly" like the rest. We deployed the 2.3.24 version of this library to production on Thursday last week (i.e. not the patch that you provided us on this issue) and are waiting to see if the issue occurs again. So far it hasn't occurred, but we probably need to wait a couple of weeks to make a more concrete assessment of whether the problem has gone away or not.

Interestingly, upgrading all of the other JARs to 2.3.24 reduced in the incidence of the issue happening quite noticeably, even though the fix for #1820 is to connection-pool.

I will keep you posted.

Cheers,

Mark

[arinban]

@glassfishrobot Commented mark.sim-smith said: Hi Ryan,

I can confirm that since we deployed 2.3.24 of connection-pool, we have not had an incident relating to this issue. You can close this issue - many thanks for your help.

Cheers,

Mark

[arinban]

@glassfishrobot Commented @rlubke said: Works properly using later versions of Grizzly.

[arinban]

@glassfishrobot Commented This issue was imported from java.net JIRA GRIZZLY-1876

[arinban]

@glassfishrobot Commented Marked as cannot reproduce on Monday, February 13th 2017, 9:58:50 am