Streaming Uploads

[simrankedia]

Hi,

I want to implement a streaming file upload (POST request) using surf. This is a huge file and I get one chunk of the file at a time. Is there a way to open an InputStream with the surf request builder and append bytes as and when received?

I want the client to not wait to receive to receive all the bytes but instead start sending chunks as soon as the bytes are received.

[jbr]

Bodies can be constructed from any AsyncBufRead: https://docs.rs/http-types/2.11.0/http_types/struct.Body.html#method.from_reader. Depending on how you receive the chunks of the file, it may already be AsyncRead and/or AsyncBufRead, but if not, you'd need some way of either providing backpressure on the chunks or accumulating them in a buffer and exposing the data as an AsyncRead/AsyncBufRead that can be owned by the request. Where's the data coming from, and is it already async?

[simrankedia]

Thanks for answering the question!

I am receiving the bytes async - every time more data is available a callback function is called which is supposed to send additional bytes over HTTP.

I am not very sure how AsyncBufRead works, would it be possible for you to provide some code pointers on how the body should be constructed in this case?

[jbr]

is it async in the rust sense, like is the source a futures or async-std type or some other type that implements AsyncRead? If not, there are a lot of possible adapters to handle this but I can't strongly personally recommend any of them, maybe others can jump in with personal experience. One possibility is writing bytes into a https://docs.rs/sluice/0.5.4/sluice/.

[simrankedia]

No the source is not async in rust sense, just that we get the bytes asynchronously.

Thanks, I will look at the adapters and figure out how to fit it into the scenario. Thanks for this. :)