search

teragrep / cfe_35

Record Router
GNU Affero General Public License v3.0
0 stars 2 forks source link

Flooding using rlp_10 occasionally causes Cannot return wrapped ByteBuffer, BufferLease phaser was already terminated! #10

Open StrongestNumber9 opened 3 weeks ago

StrongestNumber9 commented 3 weeks ago

Describe the bug

13:58:00.688 [pool-2-thread-3] ERROR com.teragrep.net_01.channel.context.IngressImpl - run() threw
java.lang.IllegalStateException: Cannot return wrapped ByteBuffer, BufferLease phaser was already terminated!
        at com.teragrep.net_01.channel.buffer.BufferLeaseImpl.buffer(BufferLeaseImpl.java:84) ~[cfe_35.jar:0.0.1-SNAPSHOT]
        at com.teragrep.rlp_03.frame.FrameClockLeaseful.submit(FrameClockLeaseful.java:67) ~[cfe_35.jar:0.0.1-SNAPSHOT]
        at com.teragrep.rlp_03.frame.FrameDelegationClock.advance(FrameDelegationClock.java:76) ~[cfe_35.jar:0.0.1-SNAPSHOT]
        at com.teragrep.net_01.channel.context.IngressImpl.run(IngressImpl.java:120) ~[cfe_35.jar:0.0.1-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

Causes rlp_10 sending thread to stall

Expected behavior

Works normally

How to reproduce

Start cfe_35 server, targetsLocalTest.json just points everything to local rlp_08 server java -Dlog4j2.configurationFile=file:src/test/resources/log4j2.xml -DserverThreads=8 -Dcfe07LookupPath=src/test/resources/cfe_07 -Dcfe16LookupPath=src/test/resources/cfe_16 -Dkin02LookupPath=src/test/resources/kin_02 -DroutingTargetsConfig=src/test/resources/targets.json -DlistenPort=4600 -jar target/cfe_35.jar

Start rlp_08 server java -DmetricsInterval=60 -Dthreads=8 -jar target/rlp_08-jar-with-dependencies.jar

Start rlp_10 towards cfe_35 java -Dthreads=32 -DmaxRecordsSent=1000000 -Dport=4600 -jar target/rlp_10-0.0.1-SNAPSHOT-jar-with-dependencies.jar

Screenshots

Software version

88ff40b9086c0d02db7a7a85e5158bff6b763b23 of branch to-9 in kortemik/cfe_35 repo

Desktop (please complete the following information if relevant):

Additional context

StrongestNumber9 commented 5 days ago

Extra logs

Closed already by: java.lang.Exception: Closed now
java.lang.Exception: Closed now
        at com.teragrep.net_01.channel.buffer.BufferLeaseImpl$ClearingPhaser.onAdvance(BufferLeaseImpl.java:136)
        at java.base/java.util.concurrent.Phaser.doArrive(Phaser.java:395)
        at java.base/java.util.concurrent.Phaser.arriveAndDeregister(Phaser.java:650)
        at com.teragrep.net_01.channel.buffer.BufferLeaseImpl.removeRef(BufferLeaseImpl.java:107)
        at com.teragrep.rlp_03.frame.RelpFrameLeaseful.close(RelpFrameLeaseful.java:109)
        at com.teragrep.rlp_03.frame.RelpFrameAccess.close(RelpFrameAccess.java:115)
        at com.teragrep.cfe_35.router.MessageParser.lambda$accept$2(MessageParser.java:193)
        at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
        at com.teragrep.rlp_03.client.TransactionService.complete(TransactionService.java:125)
        at com.teragrep.rlp_03.client.RelpClientDelegate.accept(RelpClientDelegate.java:81)
        at com.teragrep.rlp_03.frame.FrameDelegationClock.delegateFrame(FrameDelegationClock.java:114)
        at com.teragrep.rlp_03.frame.FrameDelegationClock.advance(FrameDelegationClock.java:93)
        at com.teragrep.net_01.channel.context.IngressImpl.run(IngressImpl.java:120)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
<<<
14.10.2024 14:42:32.189 [ERROR] [com.teragrep.net_01.channel.context.IngressImpl] [pool-2-thread-3] run() threw java.lang.IllegalStateException: Cannot return wrapped ByteBuffer, BufferLease phaser was already terminated!
        at com.teragrep.net_01.channel.buffer.BufferLeaseImpl.buffer(BufferLeaseImpl.java:88)
        at com.teragrep.rlp_03.frame.FrameClockLeaseful.submit(FrameClockLeaseful.java:67)
        at com.teragrep.rlp_03.frame.FrameDelegationClock.advance(FrameDelegationClock.java:76)
        at com.teragrep.net_01.channel.context.IngressImpl.run(IngressImpl.java:120)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)