search

conda / conda-package-streaming

An efficient library to read from new and old format .conda and .tar.bz2 conda packages.
https://conda.github.io/conda-package-streaming/
Other
10 stars 9 forks source link

Force ZIP64 streams #71

Closed scw closed 5 months ago

scw commented 9 months ago

If the archive stream is >2GB, currently the conversion will fail because while the conda_file object will support ZIP64 by default, open still requires an explicit call when the input file size is unknown.

Description

Prior to this change, if trying to convert an archive which exceeds 2GB (see a few packagesfrom the deep learning space), the conversion will fail with a message like:

failed files:
{'input.tar.bz2': 'File size unexpectedly exceeded ZIP64 limit'}

This ensures that the output stream is written as ZIP64 and the 2GB barrier isn't encountered for the archive contents.

dholth commented 5 months ago

Nice idea. Should we check https://github.com/conda/conda-package-streaming/pull/71/files#diff-dc2e8c2bcb91b561ad39dc1a7bc7571592ce4fd273eb8678cb45cafbe651a583L88-L89 (oh, that would be larger than the zstd-compressed data inside the zip)

Do we avoid this problem when creating a .conda directly in conda-package-handling?

dholth commented 5 months ago

Replaced with #79 , #80

dholth commented 5 months ago

Need to check with direct .conda creation also https://github.com/conda/conda-package-handling/issues/248