durch / rust-s3

Rust library for interfacing with S3 API compatible services
MIT License
498 stars 195 forks source link

put_object_stream reads stream to determine it's size #350

Open annmuor opened 1 year ago

annmuor commented 1 year ago

Describe the bug When using async put_object_stream I assume that the stream will be readed part by part as in async mode, but in reality it reads the whole stream to determine the size first. For large files it leads to OOM errors.

To Reproduce

let (mut rx, mut tx) = tokio::io::duplex(65535); // tokio duplex
tokio::spawn(async move { loop  { tx.write_all(&[1,3,4,5,6]).await; } });
Bucket::put_object_stream(b, &mut rx, "/123.txt").await;

Expected behavior Memory is not overflowing, data is written as you go.

Environment

Additional context

robinfriedli commented 1 year ago

This seems to be covered by #346