nim-lang / zip

zip wrapper for Nim
MIT License
51 stars 36 forks source link

allow passing flags to zip_open via open #48

Open brentp opened 4 years ago

brentp commented 4 years ago

AFAICT, files larger than 4.2 GB are not actually supported because we need to pass: ZIP_CM_DEFLATE64 to zip_open, but that's not possible via zip.open

I can open a PR changing:

proc open*(z: var ZipArchive, filename: string, mode: FileMode = fmRead): bool =

to

proc open*(z: var ZipArchive, filename: string, mode: FileMode = fmRead, flags:int32=0): bool =

here is the code I am using to check the problem. should fail for any file > 4.2GB.

import zip/zipfiles
import zip/libzip

var z:ZipArchive

# 5 GB zip file
var path  = "gnomad.v3.genome.zip"

if not z.open(path):
  # does not open and no way ? to get err value
  echo "couldn't open"

import zip/libzip

var err:int32
var flags:int32

flags = ZIP_CM_DEFLATE64

var w = zip_open(path, 0, addr(err))
# gives not a zip archive
echo err

err = 0
w = zip_open(path, flags, addr(err))
# gives 0
echo err
brentp commented 4 years ago

just adding that flag causes zip_open not to error, but it doesn't resolve the underlying issue. not sure how to fix.

Araq commented 4 years ago

can't we always add this flag in the implementation? Why expose this silly non-sense to zip's users.

brentp commented 4 years ago

that seems fine, but I can't get it to work even with that flag. the file will open, but I can't extract. I'll look into it more.

brentp commented 4 years ago

http://home.chpc.utah.edu/~u6000771/gnomad.v3.genome.zip

here is a test archive. the first file in the archive has a compressed size > 4.2GB. which i think is creating additional problems.

bung87 commented 4 years ago

https://github.com/nih-at/libzip/blob/178c445ef82beae9670ca68668865082eb0e1e91/lib/zip.h#L157 ZIP_CM_DEFLATE64 commented as Reserved for Tokenizing compression algorithm so may not the right use here