search

zio / zio-http

A next-generation Scala framework for building scalable, correct, and efficient HTTP clients and servers
https://zio.dev/zio-http
Apache License 2.0
781 stars 391 forks source link

Client does not handle broken servers #2383

Open erikvanoosten opened 1 year ago

erikvanoosten commented 1 year ago

When a HTTP server accepts the connection, streams back a response header (or at least some bytes) but then stops before sending a body, zio-http client (3.0.0-RC2+47-60c31e2b-SNAPSHOT) doesn't do anything any more (dead-lock?).

This server behavior was observed in production after replacing the zio-kafka HTTP client with the JVM's HTTP client. The server we need to talk to has the described behavior roughly every 10 minutes. Since this is only correlation, not causation, it is possible that behavior of the zio-kafka client has a different cause.

To reproduce one can use an intentionally evil HTTP server. For example https://github.com/kiewic/heyhttp. Such a server is also great for unit testing.

Instead of deadlocking, I expect the client to timeout after some time and replace the connection in the pool.

jdegoes commented 1 year ago

If you use ZClientAspect.timeout, do you get the behavior you are looking for? It's also possible we could add a ZClientAspect.retryAfter(duration) to address this issue.

erikvanoosten commented 1 year ago

I was not aware of that option. We did try ZIO.timeout, but that did not trigger. In addition, all other uses of the http-client were blocked as well, perhaps because the connection pool was exhausted (even though we did configure an idle timeout).

FVelasquezM commented 1 year ago

Not too sure, but I think this might be related to #2297

erikvanoosten commented 1 year ago

Not too sure, but I think this might be related to #2297

Not in my case. The http server I have to deal with is the GCP Cloud Run load balancer. Given its erratic behavior I am pretty sure they do not use zio-http 😄