TrueLayer / reqwest-middleware

Wrapper around reqwest to allow for client middleware chains.
Apache License 2.0
270 stars 79 forks source link

reqwest-retry: Retry from the last position via a Range request if connection fails in the middle of the body #36

Open sdroege opened 2 years ago

sdroege commented 2 years ago

Motivations

For use-cases where the whole body is received as a single, not too large blob, the current behaviour makes sense but for e.g. downloading media files it would make sense to also handle the case where the request succeeds but somewhere in the middle of the body there's a connection failure.

Solution

In that case a Range request to start at the last previously downloaded position could be performed on retry. This needs keeping track of the position and also of any Range header in the original requests.

conradludgate commented 2 years ago

This is an interesting idea. I think this should be a separate middleware since it's quite a strong requirement - eg we would need to read the body which is something we currently don't do.

Criteria: If there's the Accept-Ranges: bytes and a Content-Length specified in the response, we can enable the range loop.

While the reqwest::Response object cannot have the body re-inserted, I believe we may be able to reconstruct it by using http::Response to pass back in the chain.