search

fussybeaver / bollard

Docker daemon API in Rust
Apache License 2.0
907 stars 134 forks source link

Add upload_to_container_streaming to Container #445

Closed paul-hansen closed 3 months ago

paul-hansen commented 3 months ago

upload_to_container required loading the entire file into memory which was inefficient (sometimes impossible) for large files that can be streamed from disk or network.

Our use case is recreating containers with backups of the container's volumes stored in an S3 bucket. The backups are created with download_from_container which already supports streaming. Currently for us these volumes can be large and due to the limitations of upload_to_container would cause our demo server (with 1GiB of ram) to quickly run out of memory (especially if multiple customers tried to restore backups).

fussybeaver commented 3 months ago

Fabulous! The previous streaming implementation wrote this test: https://github.com/russelltg/bollard/blob/15ace255bd8f22d566855ab862e1cdb2e16a7cf7/tests/image_test.rs#L1014-L1044 .. which is just a roundtrip test, so it could be similar test.

fussybeaver commented 3 months ago

Fabulous.. Thank you very much!