search

jetty / jetty.project

Eclipse Jetty® - Web Container & Clients - supports HTTP/2, HTTP/1.1, HTTP/1.0, websocket, servlets, and more
https://eclipse.dev/jetty
Other
3.84k stars 1.91k forks source link

Client: Some HTTP/2 requests are never sent #11965

Closed mperktold closed 2 months ago

mperktold commented 3 months ago

Jetty version(s) Jetty 12.0.10

Jetty Environment core

Java version/vendor (use: java -version) openjdk version "21.0.3" 2024-04-16 LTS OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

OS type/version Windows 11

Description When using the Jetty 12 HttpClient to send a request to certain servers via HTTP/2, the request never completes. In fact, the request is only queued, but sending the request never even begins.

This only happens with some servers, which do support HTTP/2.

How to reproduce?

var connector = new ClientConnector();
connector.setSslContextFactory(new SslContextFactory.Client());
var httpClient = new HttpClient(new HttpClientTransportDynamic(connector,
    HttpClientConnectionFactory.HTTP11,
    new ClientConnectionFactoryOverHTTP2.HTTP2(new HTTP2Client(connector))
));
httpClient.start();
ContentResponse response = httpClient.newRequest("https://microsoft.sharepoint.com")
    .onRequestQueued(r -> System.out.println("request queued"))
    .onRequestBegin(r -> System.out.println("request begin"))
    .onRequestContent((r, c) -> System.out.println("request content"))
    .onRequestSuccess(r -> System.out.println("request success"))
    .onResponseBegin(r -> System.out.println("response begin"))
    .onResponseContent((r, c) -> System.out.println("response content"))
    .onResponseSuccess(r -> System.out.println("response success"))
    .send();
System.out.println(response);
httpClient.stop();

The above snippet creates a HttpClient that supports both HTTP/1.1 and HTTP2, and makes a request to "https://microsoft.sharepoint.com". Several events are logged, so we see that the request is only ever queued, but never sent.

When forcing the client to use HTTP/1.1, either by removing support for HTTP/2, or by using version(HttpVersion.HTTP_1_1), everything works as expected.

Also, the following curl command sends the same request, also using HTTP/2, and works fine: curl "https://microsoft.sharepoint.com" --http2

lorban commented 3 months ago

I created the following repository to try to reproduce this problem: https://github.com/lorban/h2-sharepoint-11965

and I get the following output which shows everything is working as expected when I run H2SharepointTest which contains your reproducer:

request queued
request begin
request success
response begin
response content
response success
HttpContentResponse[HTTP/2.0 403 null - 13 bytes]

Did I miss something?

mperktold commented 3 months ago

You're right, now it also works for me. Wow that's weird, I'm pretty sure it didn't work when I posted this. Maybe Microsoft changed something in their server.

Anyway, I found another server that does show the reported behavior: https://cashdesk-hochschwarzwald.stage.peaksolution.com Can you check with this one?

On a related note, I now also tried to replicate this with the original sharepoint URL by using Conscrypt, changing the first few lines like this:

Security.addProvider(new OpenSSLProvider());
var sslContextFactory = new SslContextFactory.Client();
sslContextFactory.setProvider("Conscrypt");
var connector = new ClientConnector();
connector.setSslContextFactory(sslContextFactory);
...

But then I get this exception:

Exception in thread "main" java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at org.eclipse.jetty.client.transport.HttpRequest.send(HttpRequest.java:707)
    at JettyIssue.main(JettyIssue.java:51)
Caused by: javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC
    at org.conscrypt.SSLUtils.toSSLHandshakeException(SSLUtils.java:361)
    at org.conscrypt.ConscryptEngine.convertException(ConscryptEngine.java:1138)
    at org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1093)
    at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:880)
    at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:751)
    at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:716)
    at org.conscrypt.Java8EngineWrapper.unwrap(Java8EngineWrapper.java:236)
    at org.eclipse.jetty.io.ssl.SslConnection.unwrap(SslConnection.java:409)
    at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.fill(SslConnection.java:742)
    at org.eclipse.jetty.io.NegotiatingClientConnection.fill(NegotiatingClientConnection.java:102)
    at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:84)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
    at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390)
    at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
    at java.base/java.lang.Thread.run(Thread.java:1583)
    Suppressed: java.io.IOException: Broken pipe
        at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.flush(SslConnection.java:1155)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:368)
        at org.eclipse.jetty.client.transport.internal.HttpSenderOverHTTP$HeadersCallback.process(HttpSenderOverHTTP.java:206)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
        at org.eclipse.jetty.client.transport.internal.HttpSenderOverHTTP.sendHeaders(HttpSenderOverHTTP.java:78)
        at org.eclipse.jetty.client.transport.HttpSender$ContentSender.process(HttpSender.java:541)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
        at org.eclipse.jetty.client.transport.HttpSender.send(HttpSender.java:85)
        at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.send(HttpChannelOverHTTP.java:86)
        at org.eclipse.jetty.client.transport.HttpChannel.send(HttpChannel.java:142)
        at org.eclipse.jetty.client.transport.HttpConnection.send(HttpConnection.java:112)
        at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP$Delegate.send(HttpConnectionOverHTTP.java:330)
        at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.send(HttpConnectionOverHTTP.java:159)
        at org.eclipse.jetty.client.transport.HttpDestination.send(HttpDestination.java:444)
        at org.eclipse.jetty.client.transport.HttpDestination.process(HttpDestination.java:420)
        at org.eclipse.jetty.client.transport.HttpDestination.process(HttpDestination.java:375)
        at org.eclipse.jetty.client.transport.HttpDestination.send(HttpDestination.java:358)
        at org.eclipse.jetty.client.transport.HttpDestination.succeeded(HttpDestination.java:291)
        at org.eclipse.jetty.client.AbstractConnectionPool.proceed(AbstractConnectionPool.java:315)
        at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.succeeded(AbstractConnectionPool.java:615)
        at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.succeeded(AbstractConnectionPool.java:593)
        at org.eclipse.jetty.util.Promise$Wrapper.succeeded(Promise.java:195)
        at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onOpen(HttpConnectionOverHTTP.java:167)
        at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:435)
        at org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:117)
        at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:87)
        ... 15 more
Caused by: java.security.cert.CertificateException: Unknown authType: GENERIC
    at java.base/sun.security.validator.EndEntityChecker.checkTLSServer(EndEntityChecker.java:290)
    at java.base/sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:149)
    at java.base/sun.security.validator.Validator.validate(Validator.java:269)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at org.conscrypt.Platform.checkServerTrusted(Platform.java:330)
    at org.conscrypt.ConscryptEngine.verifyCertificateChain(ConscryptEngine.java:1643)
    at org.conscrypt.NativeCrypto.ENGINE_SSL_read_direct(Native Method)
    at org.conscrypt.NativeSsl.readDirectByteBuffer(NativeSsl.java:567)
    at org.conscrypt.ConscryptEngine.readPlaintextDataDirect(ConscryptEngine.java:1099)
    at org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1083)
    ... 23 more

The exception appears after "request begin" is logged. After that, nothing happens and the client seems to be waiting forever. Do I need to change some security configuration, or does the server use outdated ones?

joakime commented 3 months ago

The conscrypt folks had a discussion about Unknown authType: GENERIC

https://github.com/google/conscrypt/issues/1033

Interesting to read the other issues that reference that above issue.

Google cloud, palantir, grpc, and more all have encountered this issue. Several of them report that using the Conscrypt specific TrustStore (not the JDK one) seems to help them.

lorban commented 3 months ago

I could reproduce the problem with the https://cashdesk-hochschwarzwald.stage.peaksolution.com/ URL, and it is caused by a genuine bug in the Jetty client's TLS negotiating code.

This PR contains a fix, as well as some related improvements: https://github.com/jetty/jetty.project/pull/11999

Thanks for the report!