search

softwaremill / sttp

The Scala HTTP client you always wanted!
https://sttp.softwaremill.com
Apache License 2.0
1.45k stars 309 forks source link

HttpClientZioBackend fails with 'no statuscode in response' #1609

Open nickelsen opened 2 years ago

nickelsen commented 2 years ago

While upgrading from AsyncHttpClientZioBackend to HttpClientZioBackend, we encountered this unexpected behavior.

I've created an example here, where AsyncHttpClientZioBackend works but HttpClientZioBackend fails for the same request: https://github.com/nickelsen/sttp-example

Pask423 commented 1 year ago

It seems to be a problem of HTTP/2 support in HttpClient itself. As workaround changing version to HTTP/1.1 seems to fix the problem - below as quick way on how to do it. Next week I will try to add support for HTTP version changing in more reasonable way.

val changeVersion = (v: HttpRequest) => HttpRequest
  .newBuilder(v, (n: String, c: String) => true)
  .version(HttpClient.Version.HTTP_1_1)
  .build()

val httpClientZioBackend = HttpClientZioBackend(customizeRequest = changeVersion)
  .flatMap(s => s.send(request))
nickelsen commented 1 year ago

Thanks a lot for the workaround!

nickelsen commented 1 year ago

After moving to HttpClientZioBackend we get a ReadException due to HTTP/1.1 header parser received no bytes frequently from a couple of endpoints. Not sure if it's related to HTTP/1.1 or something unrelated in the underlying HttpClient.

nickelsen commented 1 year ago

...which is actually caused by an underlying java.net.SocketException: Connection reset.

Pask423 commented 1 year ago

Could you give me some more details - stack trace, example, endpoint uri ?

nickelsen commented 1 year ago

The endpoints we see it on currently are authenticated, so I cannot give working example.

I'm also not able to reproduce in an isolated example.

We explored if this was related to our backend not being scoped (by accident), but even after we scoped it, the resets still occur.

We see the resets in about 33% of calls at the moment, but only on 1 out of the 14 endpoints we call with sttp.

We get these headers in the response:

< HTTP/1.1 200 OK
< server: envoy
< date: Wed, 09 Nov 2022 14:11:40 GMT
< content-type: application/json
< content-length: 88
< x-envoy-upstream-service-time: 31
< strict-transport-security: max-age=31536000
< Via: 1.1 google
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

The stack trace looks like this:

zio.FiberFailure: sttp.client3.SttpClientException$ReadException: Exception when sending request: POST https://gatewayapi.com/rest/mtsms?token=<token>
    at sttp.client3.impl.zio.RIOMonadAsyncError.error(RIOMonadAsyncError.scala:24)
    at sttp.client3.impl.zio.RIOMonadAsyncError.flatMap(RIOMonadAsyncError.scala:12)
    [...our code...]
Caused by: sttp.client3.SttpClientException$ReadException: Exception when sending request: POST https://gatewayapi.com/rest/mtsms?token=<token>
    at sttp.client3.SttpClientException$.defaultExceptionToSttpClientException(SttpClientException.scala:46)
    at sttp.client3.HttpClientAsyncBackend.$anonfun$adjustExceptions$1(HttpClientAsyncBackend.scala:140)
    at sttp.client3.SttpClientException$$anonfun$adjustExceptions$1.applyOrElse(SttpClientException.scala:59)
    at sttp.client3.SttpClientException$$anonfun$adjustExceptions$1.applyOrElse(SttpClientException.scala:58)
    at zio.ZIO.$anonfun$catchSome$1(ZIO.scala:356)
    at scala.util.Either.fold(Either.scala:190)
    at zio.ZIO.tryRescue$1(ZIO.scala:356)
    at zio.ZIO.$anonfun$catchSome$4(ZIO.scala:358)
    at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1149)
    at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:384)
    at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:496)
    at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:224)
    at zio.internal.FiberRuntime.run(FiberRuntime.scala:137)
    at zio.internal.ZScheduler$$anon$3.run(ZScheduler.scala:457)
Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes
    at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:348)
    at java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.onReadError(Http1Response.java:675)
    at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:302)
    at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:268)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.SocketException: Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
    at java.net.http/jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833)
    at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957)
    at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934)
Pask423 commented 1 year ago

After some checks on our side I think that is most likely a problem with HttpClient itself or something with underlying Http connection. Unfortunately I was unable to reproduce it ether with sttp or pure HttpClient.

nickelsen commented 1 year ago

Thanks a lot for looking into it! 🤩

I think we're just going to handle the resets (or whatever is going on) on our end, also given that the requests actually complete. I think that report can be ignored from the original issue.

Besides that, the upgrade from AsyncHttpClientZioBackend to HttpClientZioBackend is running smoothly. 🙌

Pask423 commented 1 year ago

FYI Since v3.8.5 there is a cleaner way to apply the workaround from above Just:

val requestWithVersion = request.httpVersion(HTTP_1_1)
val httpClientZioBackend = HttpClientZioBackend()
  .flatMap(s => s.send(requestWithVersion))
ipetkovic commented 1 year ago

@nickelsen did you perhaps manage to workaround the issue? @Pask423 any update on this one?

I have the same problem using HttpClientZioBackend. I have been using sttp client AsyncHttpClientZioBackend without issues and with upgrade to sttp client 3.8.15 I get these:

sttp.client3.SttpClientException$ReadException: Exception when sending request: POST <url> : java.net.SocketException: Connection reset.

I tried to use http 1.1 workaround but it didn't help. I cannot reproduce it locally but this happens in production app environment with many requests being sent. This happens occasionally, not always. In 1% of requests or less.

adamw commented 1 year ago

@ipetkovic this might be a separate issue (the error message is different), but maybe you can try updating Java, or checking your Java version? That's where the core implementation of the http client lives.

ipetkovic commented 1 year ago

@adamw yeah, you are probably right. I could have created separate issue. Was writing here since in one of the comments same error was mentioned. I suspect that it is related to HttpClient and not HttpClientZioBackend. But did not test it. I just went back to AsyncHttpClientZioBackend and it solved my problem so I stopped looking into it. I did not realize before that it is still available in the new version. I am using java 17.0.2.

adamw commented 1 year ago

@ipetkovic Indeed, async-http-backend was deprecated for some time, but now it's back since a new maintainer stepped in.