search

cthackers / adm-zip

A Javascript implementation of zip for nodejs. Allows user to create or extract zip files both in memory or to/from disk
MIT License
2.05k stars 375 forks source link

The problem that adm-zip reads zip files takes up a lot of memory！ #417

Open wangmings opened 2 years ago

wangmings commented 2 years ago

The problem that adm-zip reads zip files takes up a lot of memory

congzhangzh commented 2 years ago

I face the problem too, si that possible to make the get entities as a lazy iterator?

congzhangzh commented 2 years ago

It seems it already support it(do not use getEntities is a key part)?

https://github.com/cthackers/adm-zip/blob/bebbabf600a6ae5d64ee31cbc89a1274bd1d8c1f/zipFile.js#L140

https://github.com/cthackers/adm-zip/blob/bebbabf600a6ae5d64ee31cbc89a1274bd1d8c1f/zipFile.js#L148

https://github.com/cthackers/adm-zip/blob/bebbabf600a6ae5d64ee31cbc89a1274bd1d8c1f/zipFile.js#L25

@wangmings can you have a try?

wangmings commented 2 years ago

I tried to replace getEntries with getEntryCount and it couldn't be executed. I didn't study the specific source code! The process of decompressing zip files is a process from small to large! In this decompression process, a lot of operations are required, which will inevitably take up a lot of memory resources! I think this question should be unsolved!

5saviahv commented 2 years ago

Currently adm-zip loads entire file into memory, and handles things there. Yes iterators exists but they are not lazy.

melroy89 commented 1 year ago

I think there is also a memory leak present in adm-zip or memory fragmentation under Linux. Causing memory to increase every time I unzip the same file. And not giving the memory allocation back to the system. 😢

mbehr1 commented 1 year ago

I'm trying to add support for Blob instead of Buffer so that only the needed data could be put into memory with .slice(...). This should allow opening huge zip files (e.g. 50GB) as well (as long as the single files to extract are still fitting into an ArrayBuffer.)

lindstrm commented 8 months ago

I'm trying to add support for Blob instead of Buffer so that only the needed data could be put into memory with .slice(...). This should allow opening huge zip files (e.g. 50GB) as well (as long as the single files to extract are still fitting into an ArrayBuffer.)

How did this go, anything merged?

darkostanimirovic commented 1 month ago

Support for streaming would be appreciated, I'm trying to extract files > 2 GiB.

melroy89 commented 1 month ago

I moved to Go.. Problem solved 🥳 . https://gitlab.melroy.org/melroy/gitlab-artifact-deployer-go/-/blob/main/server.go?ref_type=heads#L181