search

softwaremill / sttp

The Scala HTTP client you always wanted!
https://sttp.softwaremill.com
Apache License 2.0
1.44k stars 299 forks source link

HTTP connection stuck on tcp state CLOSE_WAIT, Cats-effect backend #1924

Open altrack opened 11 months ago

altrack commented 11 months ago

Hey folks,

I'm running sttp stable 3.9.0 with the cat-effect backend with the latest JVM 17. Occasionally, the server closes the connections, and the call stuck in CLOSE_WAIT state. According to the info I found regarding this issue, it shouldn't happen and the client should close the connection.

Thanks

adamw commented 11 months ago

Is that the sttp.client3.httpclient.cats.HttpClientCatsBackend backend?

altrack commented 11 months ago

Is that the sttp.client3.httpclient.cats.HttpClientCatsBackend backend?

That's right sbt dependency com.softwaremill.sttp.client3 :: cats

package sttp.client3.httpclient.cats HttpClientCatsBackend


basicRequest
      .contentType(MediaType.ApplicationJson)
      .httpVersion(HttpVersion.HTTP_1_1)
....

Seems to have the same issue in 3.8.15

altrack commented 11 months ago

Dumping Cats-effect engine getting this suspicious trace:

cats.effect.IOFiber@118ae5d4 BLOCKED
 ? delay @ sttp.client3.impl.cats.CatsMonadError.eval(CatsMonadError.scala:19)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? flatMap @ sttp.client3.impl.cats.CatsMonadError.flatMap(CatsMonadError.scala:12)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? delay @ sttp.client3.impl.cats.CatsMonadAsyncError.$anonfun$async$1(CatsMonadAsyncError.scala:10)
 ? map @ sttp.client3.impl.cats.CatsMonadAsyncError.$anonfun$async$1(CatsMonadAsyncError.scala:10)
 ? async @ sttp.client3.impl.cats.CatsMonadAsyncError.async(CatsMonadAsyncError.scala:10)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? delay @ sttp.client3.impl.cats.CatsMonadError.eval(CatsMonadError.scala:19)
 ? flatMap @ sttp.client3.impl.cats.CatsMonadError.flatMap(CatsMonadError.scala:12)
 ? recoverWith @ sttp.client3.impl.cats.CatsMonadError.handleWrappedError(CatsMonadError.scala:17)
adamw commented 11 months ago

I think it's rather something in the java's HttpClient configuration, or the way you use the backend, rather then the specific cats integration.

Can you share some more details as to how you use the sttp client and the backend (when and how you create the backend, is it reused), and what do you mean exactly by "the call is stuck"? Does some method call block, while it shouldn't?

A reproducing example would be best, but I suspect it might be hard to create. But something close to it would help diagnose the issue.

altrack commented 11 months ago

I swapped the backend to sttp.client3.armeria.cats.ArmeriaCatsBackend and this resolved the issue. No other change in the code was done.

My use case is a batching of 4000 http requests in parallel and waiting until completion. The response from the server (Python's Uvicorn) can take up to a minute or so. After 20-40 minutes the application doesn't continue to the next batch. Entering the application pod I see some socket are not able to close. ss --tcp state CLOSE-WAIT The amount of CLOSE-WAIT connections is equal to the amount of threads in Cats-effect engine, so this entirely blocks the application. Cats-effect fiber dump for one of them is above.

Unfortunately, I don't have a test to reproduce the issue.