Write lock when sendEventBean to Esper

[kaicao]

Hi,

Please help understand what would cause long locking of WriteLock when EPEventServiceImpl#sendEventBean.

We observed that application threads are waiting for WriteLock of StatementAgentInstanceLockRW when sending event to Esper some times.

The throughput of inputting threads towards Esper decreased significantly to nearly 0 eps, and by multiple thread dumps the similar waiting for lock situation is observed as shown below:

INFO   | jvm 1    | 2022/12/01 18:34:33 | "processing-7" #238 prio=5 os_prio=0 tid=0x00007f7aac211800 nid=0x6f17 waiting on condition [0x00007f78bf486000]
INFO   | jvm 1    | 2022/12/01 18:34:33 |    java.lang.Thread.State: WAITING (parking)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at sun.misc.Unsafe.park(Native Method)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   - parking to wait for  <0x000000023520fc48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at com.espertech.esper.common.internal.context.util.StatementAgentInstanceLockRW.acquireWriteLock(StatementAgentInstanceLockRW.java:42)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at com.espertech.esper.runtime.internal.kernel.service.EPEventServiceImpl.processStatementFilterSingle(EPEventServiceImpl.java:782)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at com.espertech.esper.runtime.internal.kernel.service.EPEventServiceImpl.processMatches(EPEventServiceImpl.java:664)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at com.espertech.esper.runtime.internal.kernel.service.EPEventServiceImpl.processWrappedEvent(EPEventServiceImpl.java:453)
INFO   | jvm 1    | 2022/12/01 18:34:33 |   at com.espertech.esper.runtime.internal.kernel.service.EPEventServiceImpl.sendEventBean(EPEventServiceImpl.java:269)

Could you help us understand what can cause such WriteLock not getting unlocked for long time? If it can be statement related, what kind of statement expressions are more likely to cause such situation? So we can look into further.

Thank you!

[bernhardttom]

Typically the cause is a statement that calls Application-provided code and that code blocks, thereby locking up the statement/partition-lock. Check the application user-defined functions, extension code, static methods, event methods for blocking code.

[kaicao]

Thank you @bernhardttom !