play framework 1.4 hang during sending requests to oracle database

[karolszk]

Hello,

I have an application which causes a lot of requests to work for oracle and hangs. I don't have any idea what was reason for such behavior. (When I'm not using the app automatically is unblocked after some period of time (15-20minutes) which is unacceptable to production use. Normally application is used by 50 users.

The problem is repeatable (there are easy steps to follow by one user and go into this hang) and changing from 1.4.2 to 1.4.3 do nothing. And adding -Dorg.jboss.netty.epollBugWorkaround=true also doesn't help.

and here jstack:

"New I/O boss #27" #63 prio=5 os_prio=0 tid=0x0000000035a22000 nid=0xe70 runnable [0x000000003cd0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x000000052762cc58> (a sun.nio.ch.Util$2)
• locked <0x000000052762cc48> (a java.util.Collections$UnmodifiableSet)

• locked <0x000000052762c9d8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x000000052762c6b8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #26" #61 daemon prio=5 os_prio=0 tid=0x0000000034d52000 nid=0xaa8 runnable [0x000000003cc0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005276094a8> (a sun.nio.ch.Util$2)
• locked <0x0000000527609498> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000527609348> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000527609020> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #25" #60 daemon prio=5 os_prio=0 tid=0x0000000034d53800 nid=0xd78 runnable [0x000000003cb0f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000527621300> (a sun.nio.ch.Util$2)
• locked <0x00000005276212f0> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005276211a0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000527620e78> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #24" #59 daemon prio=5 os_prio=0 tid=0x0000000034d51800 nid=0xbb0 runnable [0x000000003ca0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000527616018> (a sun.nio.ch.Util$2)
• locked <0x0000000527616008> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000527615eb8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000527615b90> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #23" #58 daemon prio=5 os_prio=0 tid=0x0000000034d50800 nid=0x3e0 runnable [0x000000003c90f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000527614560> (a sun.nio.ch.Util$2)
• locked <0x0000000527614550> (a java.util.Collections$UnmodifiableSet)

• locked <0x000000052760f388> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x000000052760f060> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #22" #57 daemon prio=5 os_prio=0 tid=0x0000000034d53000 nid=0xbb4 runnable [0x000000003c80e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000527609f48> (a sun.nio.ch.Util$2)
• locked <0x0000000527609f38> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000527609de8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000527609ac0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #21" #56 daemon prio=5 os_prio=0 tid=0x0000000034d4d000 nid=0x13fc runnable [0x000000003c70f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000527619b00> (a sun.nio.ch.Util$2)
• locked <0x0000000527619af0> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005276199a0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000527619678> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #20" #55 daemon prio=5 os_prio=0 tid=0x0000000034d4e800 nid=0xfcc runnable [0x000000003c60e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x000000052760ea48> (a sun.nio.ch.Util$2)
• locked <0x000000052760ea38> (a java.util.Collections$UnmodifiableSet)

• locked <0x000000052760e8e8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x000000052760e5c0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #19" #54 daemon prio=5 os_prio=0 tid=0x0000000034d4d800 nid=0x1218 runnable [0x000000003c50e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x000000052761f7f8> (a sun.nio.ch.Util$2)
• locked <0x000000052761f7e8> (a java.util.Collections$UnmodifiableSet)

• locked <0x000000052761f588> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x000000052761a118> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"Hashed wheel timer #1" #53 prio=5 os_prio=0 tid=0x0000000034d50000 nid=0x718 waiting on condition [0x000000003c40e000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

• None

"play-thread-1" #52 prio=5 os_prio=0 tid=0x0000000034d4f000 nid=0x4a8 runnable [0x000000003c10d000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at oracle.net.ns.Packet.receive(Packet.java:300) at oracle.net.ns.DataPacket.receive(DataPacket.java:106) at oracle.net.ano.CryptoDataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)

• locked <0x000000053e04b6c0> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2062) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1859) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1838) at org.hibernate.loader.Loader.doQuery(Loader.java:906) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348) at org.hibernate.loader.Loader.doList(Loader.java:2550) at org.hibernate.loader.Loader.doList(Loader.java:2536) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2366) at org.hibernate.loader.Loader.list(Loader.java:2361) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1230) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at play.db.jpa.GenericModel$JPAQuery.fetch(GenericModel.java:449) at ext.Paging.fetch(Paging.java:86) at controllers.utilities.QueryByCriteriaSwitchOffFactory.getListAndSetCountPage(QueryByCriteriaSwitchOffFactory.java:202) at services.zzzzzz.OrderService.getListOrdersByCriteriaSwitchOff(OrderService.java:148) at actions.reports.SwitchOffAction.execute(SwitchOffAction.java:40) at controllers.Reports.switchOff(Reports.java:357) at sun.reflect.GeneratedMethodAccessor1908.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005274da428> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"DestroyJavaVM" #51 prio=5 os_prio=0 tid=0x0000000034d49000 nid=0xab4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:

• None

"New I/O server boss #18" #50 prio=5 os_prio=0 tid=0x0000000034d4c000 nid=0x1088 runnable [0x000000003be0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275f8360> (a sun.nio.ch.Util$2)
• locked <0x00000005275f8350> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275f7e88> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275f7b60> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #17" #49 prio=5 os_prio=0 tid=0x0000000034d4b800 nid=0x1308 runnable [0x000000003bd0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275e8890> (a sun.nio.ch.Util$2)
• locked <0x00000005275e8880> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275e36b8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275e3398> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #16" #48 prio=5 os_prio=0 tid=0x0000000034d48800 nid=0x1364 runnable [0x000000003bc0f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275f3930> (a sun.nio.ch.Util$2)
• locked <0x00000005275f3920> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275ee758> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275ee438> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #15" #47 prio=5 os_prio=0 tid=0x0000000034d4a800 nid=0xdec runnable [0x000000003bb0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275dd7f0> (a sun.nio.ch.Util$2)
• locked <0x00000005275dd7e0> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275d86b8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275d8398> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #14" #46 prio=5 os_prio=0 tid=0x0000000034d47800 nid=0xe8c runnable [0x000000003ba0f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275e2d80> (a sun.nio.ch.Util$2)
• locked <0x00000005275e2d70> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275e2c20> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275e2900> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #13" #45 prio=5 os_prio=0 tid=0x0000000034d47000 nid=0x1284 runnable [0x000000003b90e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275ede20> (a sun.nio.ch.Util$2)
• locked <0x00000005275ede10> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275edcc0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275ed9a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #12" #44 prio=5 os_prio=0 tid=0x0000000034d4a000 nid=0x1294 runnable [0x000000003b80f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275ea340> (a sun.nio.ch.Util$2)
• locked <0x00000005275ea330> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275ea1e0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275e9ec0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #11" #43 prio=5 os_prio=0 tid=0x0000000034d45800 nid=0x1068 runnable [0x000000003b70e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275df2a0> (a sun.nio.ch.Util$2)
• locked <0x00000005275df290> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275df140> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275dee20> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #10" #42 prio=5 os_prio=0 tid=0x0000000034d44800 nid=0x1164 runnable [0x000000003b60e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000005275d7d80> (a sun.nio.ch.Util$2)
• locked <0x00000005275d7d70> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000005275d7c20> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005275d7900> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O server boss #9" #41 prio=5 os_prio=0 tid=0x0000000034d46000 nid=0xe00 runnable [0x000000003b50e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a6e738> (a sun.nio.ch.Util$2)
• locked <0x0000000525a6e728> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a6e1f0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a6dc60> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #8" #40 prio=5 os_prio=0 tid=0x000000003394d000 nid=0xe94 runnable [0x000000003b40f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a8efd0> (a sun.nio.ch.Util$2)
• locked <0x0000000525a8efc0> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a8ee60> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a8ebd8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #7" #39 prio=5 os_prio=0 tid=0x0000000030a53000 nid=0x944 runnable [0x000000003b30f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a7b960> (a sun.nio.ch.Util$2)
• locked <0x0000000525a7b950> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a7b7f0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a7b430> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #6" #38 prio=5 os_prio=0 tid=0x0000000030a4f800 nid=0x670 runnable [0x000000003b20f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525adcb38> (a sun.nio.ch.Util$2)
• locked <0x0000000525adcb28> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525adc918> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525adc600> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #5" #37 prio=5 os_prio=0 tid=0x0000000030a52800 nid=0xf50 runnable [0x000000003b10e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a6ede0> (a sun.nio.ch.Util$2)
• locked <0x0000000525a6edd0> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a6d850> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a6d500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #4" #36 prio=5 os_prio=0 tid=0x0000000030a51800 nid=0x126c runnable [0x000000003b00e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a8d408> (a sun.nio.ch.Util$2)
• locked <0x0000000525a8d3f8> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a8d1e8> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a88ed0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #3" #35 prio=5 os_prio=0 tid=0x0000000030a4e800 nid=0x7b0 runnable [0x000000003af0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a87700> (a sun.nio.ch.Util$2)
• locked <0x0000000525a876f0> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a874e0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a871c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #2" #34 prio=5 os_prio=0 tid=0x0000000030a51000 nid=0xbf8 runnable [0x000000003ae0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525a75bc0> (a sun.nio.ch.Util$2)
• locked <0x0000000525a75bb0> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525a759a0> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525a75648> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"New I/O worker #1" #33 prio=5 os_prio=0 tid=0x0000000030a50000 nid=0xf1c runnable [0x000000003ad0e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x0000000525b62890> (a sun.nio.ch.Util$2)
• locked <0x0000000525b62880> (a java.util.Collections$UnmodifiableSet)

• locked <0x0000000525b62670> (a sun.nio.ch.WindowsSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x0000000525b62358> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"jobs-thread-10" #32 prio=5 os_prio=0 tid=0x0000000030a4b000 nid=0x1098 waiting on condition [0x000000003ac0e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000005b8060540> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at play.libs.F$Promise.get(F.java:60) at services.plannedWorks.registry.queue.DateRangeUpdate.update(DateRangeUpdate.java:18) at services.plannedWorks.registry.queue.UpdateQueue.tryUpdateDateRange(UpdateQueue.java:77) at services.plannedWorks.registry.queue.UpdateQueue.updateAll(UpdateQueue.java:56)
• locked <0x00000005275a88e0> (a services.plannedWorks.registry.queue.UpdateQueue) at services.plannedWorks.registry.queue.UpdateQueueJob.doJob(UpdateQueueJob.java:18) at play.jobs.Job.doJobWithResult(Job.java:54) at play.jobs.Job$2.apply(Job.java:208) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.jobs.Job.withinFilter(Job.java:185) at play.jobs.Job.call(Job.java:204) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Regards, Karol

[asolntsev]

Hi @karolszk

1. You have a very unoptimized query from database: controllers.utilities.QueryByCriteriaSwitchOffFactory.getListAndSetCountPage(QueryByCriteriaSwitchOffFactory.java:202. You need to optimize it. Probably you should add an index to database. And probably cache the result.

2. You have only one worker thread in Play pool: play-thread-1. Then this thread is busy with the slow DB query, Play cannot handle other requests. Try to increase play pool size by adding the following line in application.conf: play.pool=10

[karolszk]

Hi @asolntsev thanks for help :

ad. 1 - OK. I try to optimize it. ad. 2. I had play.pool=8

please look at the rest of the play-threads-* - I've added missing now:

"play-thread-8" #70 prio=5 os_prio=0 tid=0x0000000035a2a000 nid=0x6b4 waiting on condition [0x000000003d80e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000005274ceb60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• None

"play-thread-7" #69 prio=5 osprio=0 tid=0x0000000035a29800 nid=0x9e0 runnable [0x000000003d70d000] java.lang.Thread.State: RUNNABLE at org.hibernate.event.internal.AbstractVisitor.processValue(AbstractVisitor.java:119) at org.hibernate.event.internal.WrapVisitor.processValue(WrapVisitor.java:125) at org.hibernate.event.internal.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:76) at org.hibernate.event.internal.DefaultFlushEntityEventListener.wrapCollections(DefaultFlushEntityEventListener.java:223) at org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:162) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:251) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:100) at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:61) at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1166) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1223) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at play.db.jpa.GenericModel$JPAQuery.fetch(GenericModel.java:406) at models.Work.getEffectTags(Work.java:196) at sun.reflect.GeneratedMethodAccessor398.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196) at models.Work$$_jvstbd3.getEffectTags(Work$$_jvstb_d3.java) at services.damages.DamageService.setShowOnMapDto(DamageService.java:262) at services.damages.DamageService.addShowOnMapDetails(DamageService.java:249) at models.damages.Damage.getDamages(Damage.java:270) at models.damages.Damage.getAllDamages(Damage.java:244) at models.utilities.ExportData.DamagesToFrReport.(DamagesToFrReport.java:95) at controllers.Damages.export(Damages.java:322) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

• <0x00000005274e46a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"play-thread-6" #68 prio=5 os_prio=0 tid=0x0000000035a25000 nid=0x1120 runnable [0x000000003d60d000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at oracle.net.ns.Packet.receive(Packet.java:300) at oracle.net.ns.DataPacket.receive(DataPacket.java:106) at oracle.net.ano.CryptoDataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)

• locked <0x000000052721fdc8> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2062) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1859) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1838) at org.hibernate.loader.Loader.doQuery(Loader.java:906) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348) at org.hibernate.loader.Loader.doList(Loader.java:2550) at org.hibernate.loader.Loader.doList(Loader.java:2536) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2366) at org.hibernate.loader.Loader.list(Loader.java:2361) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1230) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at play.db.jpa.GenericModel$JPAQuery.fetch(GenericModel.java:406) at models.Work.getEffectTags(Work.java:196) at services.exportData.outage.RecordEventExportOutage.setDataReportFromWorkAttributes(RecordEventExportOutage.java:254) at services.exportData.outage.RecordEventExportOutage.(RecordEventExportOutage.java:136) at models.utilities.query.QueryDisconnectedLineOutage$3.apply(QueryDisconnectedLineOutage.java:532) at models.utilities.query.QueryDisconnectedLineOutage$3.apply(QueryDisconnectedLineOutage.java:1) at com.google.common.collect.Iterators$8.transform(Iterators.java:794) at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:301) at com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:691) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:275) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:226) at com.google.common.collect.FluentIterable.toList(FluentIterable.java:334) at models.utilities.query.QueryDisconnectedLineOutage.conversionListExportEvent(QueryDisconnectedLineOutage.java:534) at models.utilities.query.QueryDisconnectedLineOutage.getResultListExport(QueryDisconnectedLineOutage.java:464) at models.zzzzzz.DisconnectedLine.getDisconnectedObjectExport(DisconnectedLine.java:174) at services.zzzzzz.disconnectedLine.DisconnectedLineService.getListToExportData(DisconnectedLineService.java:109) at services.exportData.outage.ExportEventService.findData(ExportEventService.java:41) at services.exportData.outage.ExportEventService.export(ExportEventService.java:31) at actions.outage.exportData.ExportEventAction.execute(ExportEventAction.java:30) at controllers.Reports.exportOutageEvent(Reports.java:589) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005274cfc60> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"play-thread-5" #67 prio=5 os_prio=0 tid=0x0000000035a28000 nid=0xc1c runnable [0x000000003d50d000] java.lang.Thread.State: RUNNABLE at java.lang.String.intern(Native Method) at java.lang.Class.searchMethods(Class.java:3003) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at play.classloading.enhancers.PropertiesEnhancer$FieldAccessor.invokeReadProperty(PropertiesEnhancer.java:223) at services.zzzzzz.disconnectedLine.DisconnectedLineService.loadDisconnectedLine(DisconnectedLineService.java:129) at models.utilities.ExportData.outage.ReportsMergedOutageSimpleToFrReport.setTableBody(ReportsMergedOutageSimpleToFrReport.java:98) at models.utilities.ExportData.outage.ReportsMergedOutageSimpleToFrReport.(ReportsMergedOutageSimpleToFrReport.java:55) at controllers.Reports.exportOutageSimple(Reports.java:622) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

• <0x00000005274d88d0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"play-thread-4" #66 prio=5 os_prio=0 tid=0x0000000035a24000 nid=0x3d8 runnable [0x000000003d40c000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at oracle.net.ns.Packet.receive(Packet.java:300) at oracle.net.ns.DataPacket.receive(DataPacket.java:106) at oracle.net.ano.CryptoDataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)

• locked <0x00000005c3d45188> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2062) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1859) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1838) at org.hibernate.loader.Loader.doQuery(Loader.java:906) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:318) at org.hibernate.loader.Loader.loadEntity(Loader.java:2145) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:82) at org.hibernate.loader.entity.EntityLoader.loadByUniqueKey(EntityLoader.java:161) at org.hibernate.persister.entity.AbstractEntityPersister.loadByUniqueKey(AbstractEntityPersister.java:2374) at org.hibernate.type.EntityType.loadByUniqueKey(EntityType.java:765) at org.hibernate.type.EntityType.resolve(EntityType.java:505) at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:168) at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:137) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1112) at org.hibernate.loader.Loader.processResultSet(Loader.java:969) at org.hibernate.loader.Loader.doQuery(Loader.java:917) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:318) at org.hibernate.loader.Loader.loadEntity(Loader.java:2145) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:82) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:72) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3939) at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:462) at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:431) at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:208) at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:264) at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:152) at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1053) at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:980) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:714) at org.hibernate.type.EntityType.resolve(EntityType.java:502) at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:168) at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:137) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1112) at org.hibernate.loader.Loader.processResultSet(Loader.java:969) at org.hibernate.loader.Loader.doQuery(Loader.java:917) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348) at org.hibernate.loader.Loader.doList(Loader.java:2550) at org.hibernate.loader.Loader.doList(Loader.java:2536) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2366) at org.hibernate.loader.Loader.list(Loader.java:2361) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1230) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at play.db.jpa.GenericModel$JPAQuery.fetch(GenericModel.java:406) at ext.Paging.fetch(Paging.java:83) at models.damages.Damage.getDamages(Damage.java:260) at models.damages.Damage.getAllDamages(Damage.java:244) at models.utilities.ExportData.DamagesToFrReport.(DamagesToFrReport.java:95) at controllers.Damages.export(Damages.java:322) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005274e2e68> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"play-thread-3" #65 prio=5 os_prio=0 tid=0x0000000035a27000 nid=0x940 runnable [0x000000003d30d000] java.lang.Thread.State: RUNNABLE at org.hibernate.event.internal.AbstractVisitor.processValue(AbstractVisitor.java:123) at org.hibernate.event.internal.WrapVisitor.processValue(WrapVisitor.java:125) at org.hibernate.event.internal.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:76) at org.hibernate.event.internal.DefaultFlushEntityEventListener.wrapCollections(DefaultFlushEntityEventListener.java:223) at org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:162) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:251) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:100) at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:61) at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1166) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1223) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at dao.netEvents.NetEventDao.findByDamageId(NetEventDao.java:102) at services.netEvents.NetEventService.getByDamageId(NetEventService.java:124) at models.damages.DamageBase.getLowVoltageCircuitByWork(DamageBase.java:251) at models.damages.DamageBase.getSubstationAndObjectName(DamageBase.java:348) at models.utilities.ExportData.DamagesToFrReport.getObjectName(DamagesToFrReport.java:372) at models.utilities.ExportData.DamagesToFrReport.prepareReport(DamagesToFrReport.java:301) at models.utilities.ExportData.DamagesToFrReport.(DamagesToFrReport.java:115) at controllers.Damages.export(Damages.java:322) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

• <0x00000005275c2b98> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"play-thread-2" #64 prio=5 os_prio=0 tid=0x0000000035a23800 nid=0x694 runnable [0x000000003d20d000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at oracle.net.ns.Packet.receive(Packet.java:300) at oracle.net.ns.DataPacket.receive(DataPacket.java:106) at oracle.net.ano.CryptoDataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)

• locked <0x00000004c08ce1c0> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2062) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1859) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1838) at org.hibernate.loader.Loader.doQuery(Loader.java:906) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348) at org.hibernate.loader.Loader.doList(Loader.java:2550) at org.hibernate.loader.Loader.doList(Loader.java:2536) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2366) at org.hibernate.loader.Loader.list(Loader.java:2361) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1230) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at play.db.jpa.GenericModel$JPAQuery.fetch(GenericModel.java:449) at ext.Paging.fetch(Paging.java:86) at controllers.utilities.QueryByCriteriaSwitchOffFactory.getListAndSetCountPage(QueryByCriteriaSwitchOffFactory.java:202) at services.zzzzzz.OrderService.getListOrdersByCriteriaSwitchOff(OrderService.java:148) at actions.reports.SwitchOffAction.execute(SwitchOffAction.java:40) at controllers.Reports.switchOff(Reports.java:357) at sun.reflect.GeneratedMethodAccessor1908.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:544) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:494) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:489) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:458) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:162) at play.server.PlayHandler$NettyInvocation.execute(PlayHandler.java:326) at play.Invoker$Invocation$1.apply(Invoker.java:292) at play.Invoker$Invocation$1.apply(Invoker.java:288) at play.db.jpa.JPA.withTransaction(JPA.java:258) at play.db.jpa.JPA.withinFilter(JPA.java:217) at play.db.jpa.JPAPlugin$TransactionalFilter.withinFilter(JPAPlugin.java:298) at play.Invoker$Invocation.withinFilter(Invoker.java:271) at play.Invoker$Invocation.run(Invoker.java:288) at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:303) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

  Locked ownable synchronizers:

• <0x00000005274e3718> (a java.util.concurrent.ThreadPoolExecutor$Worker)

[karolszk]

Hi again, @asolntsev "Try to increase play pool size by adding the following line in application.conf: play.pool=10" how big may be this value? From doc number_of_processors+1 but ts it possible to increase to 50? :) Regards, Karol

[asolntsev]

Hi @karolszk ! Most of your "play-thread-*" threads are busy with database operations. So, I repeat: you need to analyze your selects and probably add some indexes to database.

I think that recommendation "doc number_of_processors+1" is only valid for applications that use asynchronous operations everywhere (like Rest services calls). In other words, threads "play-thread-*" never wait for anything in such applications.

Oracle jdbc driver is not asynochronous, meaning that your threads do wait for database response. In this case I recommend setting `play.pool=database pool + N".