Jetty threads getting blocked in response timeout testing causing application crash

[owaiscs001]

Description I'm testing a traffic scenario with TPS of ~1000, the HTTP/2 server has a high delay hence all the message send by Jetty HTTP/2 client to the server gets timed out. Below is my observation with this run.

• 1000+ Jetty threads are created
• Among them ~200 threads were in blocked state (traceback added below)
• The application becomes unresponsive at this stage and eventually dies (OOM)

Request: I encountered an uncaught exception in the system, and I'm reaching out for clarification on the underlying cause. The stacktrace attached below shows the thread is blocked while attempting to acquire a lock on the PrintStream object. Could you help clarify what might be causing this blockage? Is there a specific condition or known issue that could lead to this uncaught exception?

Root Cause Analysis: Are there any tools or methods to help identify the reasons behind the thread blocking?

Additional Information: Stacktrace of the blocked threads

`"@365a6a43-35" #35 prio=5 os_prio=0 cpu=28.09ms elapsed=689.64s tid=0x00007f9741f42d20 nid=0x3e waiting for monitor entry [0x00007f970c8fa000] java.lang.Thread.State: BLOCKED (on object monitor) at java.io.PrintStream.write(java.base@17.0.7/PrintStream.java:695)

• waiting to lock <0x0000000680c98068> (a java.io.PrintStream) at java.io.PrintStream.print(java.base@17.0.7/PrintStream.java:863) at java.lang.ThreadGroup.uncaughtException(java.base@17.0.7/ThreadGroup.java:1084) at java.lang.ThreadGroup.uncaughtException(java.base@17.0.7/ThreadGroup.java:1077) at reactor.core.scheduler.Schedulers.handleError(Schedulers.java:1208) at reactor.core.scheduler.ExecutorScheduler.schedule(ExecutorScheduler.java:70) at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.trySchedule(MonoPublishOn.java:146) at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.onNext(MonoPublishOn.java:120) at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) at reactor.core.publisher.Mono.subscribe(Mono.java:4400) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onError(FluxTimeout.java:219) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2063) at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544) at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) at reactor.core.publisher.Operators.error(Operators.java:198) at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:56) at reactor.core.publisher.Mono.subscribe(Mono.java:4400) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278) at org.eclipse.jetty.reactive.client.internal.AbstractSingleProcessor.onError(AbstractSingleProcessor.java:120) at org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.onComplete(ResponseListenerProcessor.java:141) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:213) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:205) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:477) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:457) at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:553) at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:149) at org.eclipse.jetty.client.HttpSender.terminateRequest(HttpSender.java:298) at org.eclipse.jetty.client.HttpSender.abortRequest(HttpSender.java:280) at org.eclipse.jetty.client.HttpSender.abort(HttpSender.java:385) at org.eclipse.jetty.client.HttpSender.lambda$executeAbort$0(HttpSender.java:252) at org.eclipse.jetty.client.HttpSender$$Lambda$2278/0x0000000801a3a400.run(Unknown Source) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:894) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1038) at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)`

Your Environment

• Reactor version(s) used: 3.4.18
• Other relevant libraries versions (eg. netty, ...): Jetty 11.0.11
• JVM version (java -version): 17.0.7
• OS and version (eg uname -a): Linux

[chemicL]

Hi,

I have a couple points:

1. BlockHound can be used to track blocking calls.
2. From the stacktrace - an onError is triggered and it struggles to delegate to a provided Scheduler based on a wrapped Executor. That Executor has a Thread that's within a ThreadGroup, which has a defined uncaughtExceptionHandler. My guess is that the actual handler is Schedulers.defaultUncaughtException, which delegates to slf4j to log the issue. It looks as if in this case you're using a blocking slf4j backend - perhaps Logback's ConsoleAppender or something of that sort? If that's the case it's best to wrap it with an AsyncAppender or an equivalent in a different implementation.
3. reactor-core 3.4 is out of open-source support, please consider upgrading.
4. We don't maintain the Jetty HttpClient, perhaps it's worth reaching out in the respective repository if you end up discovering an issue is caused by its implementation.

In case there's a genuine issue in reactor-core, please reopen with a relevant reproducer. Good luck on your quest to uncover the issue! :)