search

scosman / zipstreamer

Zip File Streaming Microservice - stream zip files on the fly
MIT License
119 stars 18 forks source link

Support for range requests? #32

Closed pcriv closed 1 year ago

pcriv commented 1 year ago

Does zipstreamer support requesting partial ranges?

scosman commented 1 year ago

Short answer: nope, unfortunately not!

Long answer: No, and supporting it would be a major effort (more than a re-write, a new bigger project).

Zipstreamer can handle so many parallel streams because it doesn't have all the files for all the files on disk -- it streams them. The zip is built on the fly, and the index is written at the end. It doesn't even support the content-length header because we don't know how large the file will be when we start streaming it.

A range request would require being able to pick up where it left off, which implies a static zip file or a completely deterministic way of building files. The first would negate the main idea behind zipstreamer (on the fly stream without needing disk/state). The second might be possible... but would be a huge amount of work, can't be done with the standard zip library, and to be certain you were getting it deterministic would need to re-process all the files (not particularly efficient). You'd also need some long-lived IDs, which is also kinda against the idea of on-the-fly streaming service.

If that's really needed, I think you want something more like Zipstreamer -> S3 cache -> serve file. S3 supports range requests natively. But that caching layer wouldn't be in the scope of this project.

pcriv commented 1 year ago

Thanks a lot for the detailed response @scosman !