search

jetty / jetty.project

Eclipse Jetty® - Web Container & Clients - supports HTTP/2, HTTP/1.1, HTTP/1.0, websocket, servlets, and more
https://eclipse.dev/jetty
Other
3.82k stars 1.91k forks source link

Jetty Http/2 Client send operation getting stuck in performance run #12021

Open kdtech1001 opened 2 months ago

kdtech1001 commented 2 months ago

Jetty version(s) 11.0.20 Java version/vendor (use: java -version) 17.0.11 Spring boot version(s) 2.7.21 OS type/version Linux

I know Jetty 11 is out of support and we are migrating to 12. but in one of our old release we are observing that Jetty send is getting stuck for a good amount of time while running performance (4K TPS) on Ipv6 using Jetty HTTP/2 Client. Everything works if we use Ipv4. We are using jetty with reactor core to make asynchronous send.

Thread Stacktrace

java.base@17.0.11/jdk.internal.misc.Unsafe.park(Native Method), 
java.base@17.0.11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211), 
java.base@17.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715), java.base@17.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:938), 
java.base@17.0.11/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153), 
java.base@17.0.11/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322), 
java.base@17.0.11/java.net.InMemoryCookieStore.get(InMemoryCookieStore.java:120), 
org.eclipse.jetty.client.HttpConnection.normalizeRequest(HttpConnection.java:224), 
org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.normalizeRequest(HttpConnectionOverHTTP2.java:158), 
org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:111), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:440), 
org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:416), 
org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:371), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:354), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:348), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:325), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:304), 
org.eclipse.jetty.client.HttpClient.send(HttpClient.java:501), 
org.eclipse.jetty.client.HttpRequest$$Lambda$1511/0x0000000800c886b8.accept(Unknown Source), 
org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:879), 
org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:867), 
org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.send(ResponseListenerProcessor.java:169), 
org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.onRequest(ResponseListenerProcessor.java:155), 
org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.request(AbstractSinglePublisher.java:90), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:134), 
org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.subscribe(AbstractSinglePublisher.java:62), 
reactor.core.publisher.MonoSource.subscribe(MonoSource.java:71), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203), 
reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126), 
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:250), 
reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:324), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:209), 
reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89), 
reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:121), 
reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60), 
reactor.core.publisher.MonoFromFluxOperator.subscribe(MonoFromFluxOperator.java:81), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203), 
reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:140), 
org.springframework.http.client.reactive.JettyClientHttpRequest.lambda$writeWith$2(JettyClientHttpRequest.java:98), 
org.springframework.http.client.reactive.JettyClientHttpRequest$$Lambda$2062/0x0000000800ebd3c0.accept(Unknown Source), 
reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58), 
reactor.core.publisher.Mono.subscribe(Mono.java:4495), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263), 
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51), 
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165), 
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74), 
reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2811), 
reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180), 
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152), 
reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2547), 
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171), 
reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103), 
reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2878), 
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341), 
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2215), 
reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115), 
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96), 
reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68), 
reactor.core.publisher.Mono.subscribe(Mono.java:4495), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263), 
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51), 
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64), 
reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53), 
reactor.core.publisher.Mono.subscribe(Mono.java:4495), 
reactor.core.publisher.Mono.subscribeWith(Mono.java:4561), 
reactor.core.publisher.Mono.subscribe(Mono.java:4462), 
reactor.core.publisher.Mono.subscribe(Mono.java:4398), 
reactor.core.publisher.Mono.subscribe(Mono.java:4370), 
Test.Application.asynclients.SendHttpRequest.buildRequest(SendHttpRequest.java:543)

Trying to understand what could be the reason behind InMemoryCookieStore.get() getting stuck.

Any pointers would be helpful. Thanks.

sbordet commented 2 months ago

Trying to understand what could be the reason behind InMemoryCookieStore.get() getting stuck.

No idea. We did not see this issue in other IPv6 deployments we know.

This stack trace shows the thread that is waiting for the lock, so there must be another thread that has the lock -- you should look at what that thread is doing.