Cannot parse file header with uncompressed size, compressed size, or local file header offset of 0xffffffff, if not in Zip64 format

thejoshwolfe / yauzl

yet another unzip library for node

MIT License

678 stars 77 forks source link

This issue is a companion to #108, but affects a different part of the parser. yauzl 2.10.0 cannot parse zip files whose uncompressedSize, compressedSize, or relativeOffsetOfLocalHeader is 0xffffffff, unless the file is in Zip64 format. The relevant code is here: https://github.com/thejoshwolfe/yauzl/blob/02a5ca69c7713f6d2897cc02f2acc1df21093e3d/index.js#L333-L337

When yauzl sees a 0xffffffff value in one of these fields, it assumes that the zip file must be in Zip64 format. But APPNOTE.TXT 4.4.8, 4.4.9, and 4.4.16 say (emphasis mine):

If an archive is in ZIP64 format and the value in this field is 0xFFFFFFFF, the size will be in the corresponding 8 byte ZIP64 extended information extra field.

The way I interpret this statement, the logic should not be what yauzl does now:

if a value is 0xffffffff:
    parse the Zip64 extended information extra field

but should instead be:

if a Zip64 extended information extra field is found:
    replace only the values that are 0xffffffff

This issue is basically the same as https://github.com/golang/go/issues/31692. The difference is that Go archive/zip has a special-case workaround for the compressed size, but not the other two fields.

The test cases which follow are able to be parsed by Info-ZIP UnZip (unzip) and Python zipfile (python3 -m zipfile -e). Go archive/zip can parse the uncompressed size test but not the other two.

Uncompressed size test case

ffffffff-uncompressedSize.zip.gz (remove 1 layer of gzip before testing)

# 65535 * 65537 = 0xffffffff
dd if=/dev/zero bs=65535 count=65537 of=pad
touch -d '2019-05-01 00:00:00 UTC' pad
rm -f ffffffff-uncompressedSize.zip
TZ=UTC zip -9 -X ffffffff-uncompressedSize.zip pad
gzip -9 -k ffffffff-uncompressedSize.zip

zipinfo -v says:

  uncompressed size:                              4294967295 bytes

yauzl does not parse it:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: expected zip64 extended information extra field
    at node_modules/yauzl/index.js:347:46
    at node_modules/yauzl/index.js:631:5
    at node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

Compressed size test case

It's difficult to construct a test case that has compressedSize = 0xffffffff and uncompressedSize < 0xffffffff. It could plausibly happen if the compressor implements "store" mode not as compressionMethod 0, but as compressionMethod 8 (DEFLATE) and non-compressed blocks, which increase the compressed size slightly. The example here instead has compressedSize = uncompressedSize = 0xffffffff.

ffffffff-compressedSize.zip.gz.gz (remove 2 layers of gzip before testing)

# 65535 * 65537 = 0xffffffff
dd if=/dev/zero bs=65535 count=65537 of=pad
touch -d '2019-05-01 00:00:00 UTC' pad
rm -f ffffffff-compressedSize.zip
TZ=UTC zip -0 -X ffffffff-compressedSize.zip pad

zipinfo -v says:

  compressed size:                                4294967295 bytes
  uncompressed size:                              4294967295 bytes

yauzl does not parse it:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: expected zip64 extended information extra field
    at node_modules/yauzl/index.js:347:46
    at node_modules/yauzl/index.js:631:5
    at node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

Local file header offset test case

ffffffff-relativeOffsetOfLocalHeader.zip.gz.gz (remove 2 layers of gzip before testing)

# 216186 * 19867 = 0xffffffff - len("pad") - 30
dd if=/dev/zero bs=216186 count=19867 of=pad
echo test > test.txt
touch -d '2019-05-01 00:00:00 UTC' pad test.txt
rm -f ffffffff-relativeOffsetOfLocalHeader.zip
TZ=UTC zip -0 -X ffffffff-relativeOffsetOfLocalHeader.zip pad test.txt

zipinfo -v says:

  offset of local header from start of archive:   4294967295
                                                  (00000000FFFFFFFFh) bytes

yauzl does not parse it:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: expected zip64 extended information extra field
    at node_modules/yauzl/index.js:347:46
    at node_modules/yauzl/index.js:631:5
    at node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

josh@nixos:~/dev/yauzl$ nix-run --pure -p nodejs -- node examples/compareCentralAndLocalHeaders.js ~/tmp/asdf/ffffffff-compressedSize.zip pad ┌───────────────────────────┬──────────┬──────────┬────┐ │field │ central│ local│diff│ ├───────────────────────────┼──────────┼──────────┼────┤ │versionMadeBy │ 0x31e│ -│ │ │versionNeededToExtract │ 0xa│ 0xa│ │ │generalPurposeBitFlag │ 0x0│ 0x0│ │ │compressionMethod │ 0x0│ 0x0│ │ │lastModFileTime │ 0x0│ 0x0│ │ │lastModFileDate │ 0x4ea1│ 0x4ea1│ │ │crc32 │ 0x0│ 0x0│ │ │compressedSize │0xffffffff│0xffffffff│ │ │uncompressedSize │0xffffffff│0xffffffff│ │ │fileNameLength │ 0x3│ 0x3│ │ │extraFieldLength │ 0x0│ 0x0│ │ │fileCommentLength │ 0x0│ -│ │ │internalFileAttributes │ 0x0│ -│ │ │externalFileAttributes │0x81a40000│ -│ │ │relativeOffsetOfLocalHeader│ 0x0│ -│ │ └───────────────────────────┴──────────┴──────────┴────┘ central.fileName: 706164 (local matches) central.extraField: (local matches) central.comment: josh@nixos:~/dev/yauzl$ nix-run --pure -p nodejs -- node examples/compareCentralAndLocalHeaders.js ~/tmp/asdf/ffffffff-relativeOffsetOfLocalHeader.zip pad ┌───────────────────────────┬──────────┬──────────┬────┐ │field │ central│ local│diff│ ├───────────────────────────┼──────────┼──────────┼────┤ │versionMadeBy │ 0x31e│ -│ │ │versionNeededToExtract │ 0xa│ 0xa│ │ │generalPurposeBitFlag │ 0x0│ 0x0│ │ │compressionMethod │ 0x0│ 0x0│ │ │lastModFileTime │ 0x0│ 0x0│ │ │lastModFileDate │ 0x4ea1│ 0x4ea1│ │ │crc32 │0xf86bd088│0xf86bd088│ │ │compressedSize │0xffffffde│0xffffffde│ │ │uncompressedSize │0xffffffde│0xffffffde│ │ │fileNameLength │ 0x3│ 0x3│ │ │extraFieldLength │ 0x0│ 0x0│ │ │fileCommentLength │ 0x0│ -│ │ │internalFileAttributes │ 0x0│ -│ │ │externalFileAttributes │0x81a40000│ -│ │ │relativeOffsetOfLocalHeader│ 0x0│ -│ │ └───────────────────────────┴──────────┴──────────┴────┘ central.fileName: 706164 (local matches) central.extraField: (local matches) central.comment: test.txt ┌───────────────────────────┬──────────┬──────────┬────┐ │field │ central│ local│diff│ ├───────────────────────────┼──────────┼──────────┼────┤ │versionMadeBy │ 0x31e│ -│ │ │versionNeededToExtract │ 0xa│ 0xa│ │ │generalPurposeBitFlag │ 0x0│ 0x0│ │ │compressionMethod │ 0x0│ 0x0│ │ │lastModFileTime │ 0x0│ 0x0│ │ │lastModFileDate │ 0x4ea1│ 0x4ea1│ │ │crc32 │0x3bb935c6│0x3bb935c6│ │ │compressedSize │ 0x5│ 0x5│ │ │uncompressedSize │ 0x5│ 0x5│ │ │fileNameLength │ 0x8│ 0x8│ │ │extraFieldLength │ 0x0│ 0x0│ │ │fileCommentLength │ 0x0│ -│ │ │internalFileAttributes │ 0x0│ -│ │ │externalFileAttributes │0x81a40000│ -│ │ │relativeOffsetOfLocalHeader│0xffffffff│ -│ │ └───────────────────────────┴──────────┴──────────┴────┘ central.fileName: 746573742e747874 (local matches) central.extraField: (local matches) central.comment:

thejoshwolfe / yauzl