Stream connection hanging forever

[jeanp413]

Describe the bug

I have a stream service in a server behind a proxy that has a timeout of 10 min and then closes the connection, when this happens I expect the async iterator to throw an exception (so I can reconnect again) but instead it just hangs forever.

I found this related issue in the grpc-js library which looks the same, but I've been able to workaround it by enabling sending keepalive pings with 'grpc.keepalive_time_ms' option, can a similar option be added to connect-node library?

[timostamm]

Thanks for raising this, Jean Pierre. Stream termination with a proxy in between can be tricky. To clarify, this happens with a client transport from @bufbuild/connect-node over H2?

Just hanging forever is not acceptable. Yes I'm sure we will be able to provide a fix for this.

[jeanp413]

yes I'm using @bufbuild/connect-node over http2

[timostamm]

@jeanp413, it took a while, but this should be addressed by https://github.com/bufbuild/connect-es/pull/619. There was a straight-forward bug in the client implementation that caused connections that unexpectedly ended to hang.

We do not send HTTP/2 PING frames to keep the session alive, but it seems that your issue is likely the bug, not the keep alive.

In case you do still experience an issue after this is released, let us know.

[jeanp413]

hey @timostamm just gave a quick try from sources and still can repro, but seems that the issue is not related to the proxy I mention initially, the thing is my stream service does not sent data continuously only when some event happens, so it could be that it doesn't send any data for 10 min but after that time the socket is not responsive anymore and the async iterator hangs forever, found this stackoverflow question that mentions sending keepalive pings too

[timostamm]

Thanks for testing it, @jeanp413! I'm surprised to hear that it's still happening. Could you share some information about the server and your deployment?

I have created https://github.com/bufbuild/connect-es/issues/627 specifically for HTTP/2 PING frames, and will leave this issue closed. We're currently adding client-side deadlines, so that a timeout always aborts the request without relying on the server to close. This should also help to deal with such an issue, although I'm aware it doesn't replace actively monitoring the connection.

[timostamm]

I forgot to mention: Talking about HTTP/2 ping to others, I've also heard about issues with especially in combination with proxies that might terminate ack the pings for your client, while your upstream connection dies due to idle. They are not a silver bullet either.

[timostamm]

@jeanp413, we have implemented keep-alive with https://github.com/bufbuild/connect-es/pull/673. You should be able to set pingIntervalMs, which behaves very similar to grpc.keepalive_time_ms.

[jeanp413]

Awesome, will try it out early next week, thanks :tada: