add support for splitting zips greater than 500 gigs - Githubissues

kieranjol / IFIscripts

Detailed documentation is available here: http://ifiscripts.readthedocs.io/en/latest/index.html

http://ifiscripts.readthedocs.io/en/latest/index.html

MIT License

50 stars 34 forks source link

add support for splitting zips greater than 500 gigs #360

Closed kieranjol closed 5 years ago

kieranjol commented 5 years ago

This will split up zips that are greater than 500 gigs in size. Merging is blocked because the internal python Zipfile module does not support split zip archives, and we use Zipfile to perform the validation. So 7za validation is needed instead. This requires:

Parsing the 7za error messages to detect success and failure
Figuring out if we should just use 7za for all verification. I'd imagine that the potential of multithreading could boost performance in general.