search

krolaw / zipstream

Reads zip files from io.Reader
Other
55 stars 16 forks source link

How can it help if the zip file itself need to be loaded into memory first? #6

Closed onkarbanerjee closed 1 month ago

onkarbanerjee commented 1 month ago

I am currently trying to download zips from an S3 bucket, unzip and upload the individual contents. However a single zip can go as large as 2GBs and i dont want to load that into memory so was looking for a library that helps to stream from the zip, so needed something that takes an io.Reader instead of io.ReaderAt. But the problem is that it can only work well if the zip file is available in it's entirety, right? If the io.Reader provided to the zipstream.NewReader() reads it in chunks, it wouldn't work no? Then what would really be the point of streaming from a zip in anycase, is what I am wondering.

krolaw commented 1 month ago

While the index is at the end of the zip file, there are divider markers between the stored files, allowing extraction of desired files without loading the entire archive into memory.