search

JuliaCloud / AWSS3.jl

AWS S3 Simple Storage Service interface for Julia.
Other
49 stars 34 forks source link

Support for streaming `S3Path` data #204

Open omus opened 3 years ago

omus commented 3 years ago

After running into https://github.com/JuliaCloud/AWSS3.jl/issues/197 I discovered that when using an S3Path as a I/O stream what is actually occurring is that a FileBuffer is being used which will load the entire S3 contents into an IOBuffer the first time any bytes are read.

AWS S3 supports the "Range" URI request parameter which allows request byte ranges. Effectively this allows use to perform a multi-part download of the file which should allow us to download chunks in parallel and also allow us to read a portion of the file without having to download all of the content. See the multi-part upload documentation for a list of advantages that should mostly also apply to downloading.

omus commented 3 years ago

Related #186

omus commented 2 years ago

Actively working on this.

omus commented 2 years ago

Ran into an issue while developing this: https://github.com/JuliaCloud/AWS.jl/issues/536