Hang after request has begun, but before the message is received off the wire

[stuhood]

Bug Report

Version

tonic 0.8.2
tonic-build 0.8.2

Platform

Linux remoting-storage-server-85dcb67bbc-r92pc 5.4.209-116.367.amzn2.x86_64 #1 SMP Wed Aug 31 00:09:52 UTC 2022 x86_64 GNU/Linux

Description

When a concurrency_limit_per_connection is set and exceeded (i.e., set to 64, and an InFlightRequests layer shows 64 requests in flight), I have a reproducible situation where a server hangs in: https://github.com/hyperium/tonic/blob/33e22bbc5ef1b74de82394c3ebfea27382419620/tonic/src/server/grpc.rs#L287-L290 ... waiting for a command to arrive off the wire.

Given that this is the server waiting for the client (which is also a tonic service) to do something, it seems likely that the client is to blame in some way... but hanging after having received the headers of the request, but before the request message has arrived felt very suspicious.

I hesitated to file this because I don't have enough information to reproduce it in isolation, but:

• it is unique enough that it's possible that there is still a useful diagnosis to be made
• I can repro it, and so if you have any ideas about where to look in the client, that would be helpful

[LucioFranco]

So the limit stuff just works ontop of hyper so maybe it should be possible to see if we can reproduce it there? So it could maybe have something to do with how poll_ready is handled within hyper.