benoitc / hackney

simple HTTP client in Erlang
Other
1.34k stars 427 forks source link

Fix intermittent chunked response hang #746

Open danielfinke opened 1 month ago

danielfinke commented 1 month ago

When streaming a chunked response, it is possible to cause a TCP receive hang under particular circumstances. If at one point the parser buffer doesn't have the whole chunk, at a later point the buffer ends up empty <<>>, and subsequently hackney_response:stream_body/1 is called, hackney_response:recv/2 will hang if the expected remaining size exceeds the remainder of the response. That expected size is actually stale, from the earlier point when the parser did not have the whole chunk. This issue slipped in with benoitc/hackney#710.

This was identified when using https://github.com/benoitc/couchbeam and sending several chunked requests. If the last non-terminating chunk completed the response JSON object, hackney_response:skip_body/1 is called to discard the remaining body, but is told to receive a number of bytes equal to the expected remaining size which will frequently exceed the small terminating chunk and trailers. As a result, the recv operation hangs waiting for bytes that will never arrive.

Now, the transfer state (BufSize/ExpectedSize) are reset after each successful chunk. The speed benefit of benoitc/hackney#710 is retained (tested with the same approach as in that PR).

jamesaimonetti commented 4 weeks ago

@benoitc this has been working well for us lately. Curious if you have a chance to review?

benoitc commented 2 weeks ago

on it sorry for the delay.