unsupported compression method: 9

lurch commented 7 years ago

We recently had a report against Etcher that a user was getting a unsupported compression method: 9 error when trying to extract a particular archive.

I downloaded the archive myself, and managed to trace the problem to yauzl. When I clone this repo and run node examples/dump.js --no-contents /home/andrew/Downloads/RPI-CNC-V400.zip I get this output:

Entry {
  versionMadeBy: 45,
  versionNeededToExtract: 45,
  generalPurposeBitFlag: 0,
  compressionMethod: 9,
  lastModFileTime: 25321,
  lastModFileDate: 18998,
  crc32: 2459691682,
  compressedSize: 2359888958,
  uncompressedSize: 8006926336,
  fileNameLength: 14,
  extraFieldLength: 12,
  fileCommentLength: 0,
  internalFileAttributes: 0,
  externalFileAttributes: 32,
  relativeOffsetOfLocalHeader: 0,
  fileName: 'RPI_CNC_V4.img',
  extraFields: [ { id: 1, data: <Buffer 00 00 40 dd 01 00 00 00> } ],
  fileComment: '',
  comment: '' }
2017-01-22T12:23:18.000Z

Without the --no-contents flag I get:

Entry {
  versionMadeBy: 45,
  versionNeededToExtract: 45,
  generalPurposeBitFlag: 0,
  compressionMethod: 9,
  lastModFileTime: 25321,
  lastModFileDate: 18998,
  crc32: 2459691682,
  compressedSize: 2359888958,
  uncompressedSize: 8006926336,
  fileNameLength: 14,
  extraFieldLength: 12,
  fileCommentLength: 0,
  internalFileAttributes: 0,
  externalFileAttributes: 32,
  relativeOffsetOfLocalHeader: 0,
  fileName: 'RPI_CNC_V4.img',
  extraFields: [ { id: 1, data: <Buffer 00 00 40 dd 01 00 00 00> } ],
  fileComment: '',
  comment: '' }
2017-01-22T12:23:18.000Z
/tmp/yauzl/examples/dump.js:25
        if (err) throw err;
                 ^

Error: unsupported compression method: 9
    at /tmp/yauzl/index.js:511:25
    at /tmp/yauzl/index.js:618:5
    at /tmp/node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:682:17)

Are there any plans for yauzl to support this compression method 9? Wikipedia says https://en.wikipedia.org/wiki/Zip_(file_format)#Compression_methods and unzip -v /home/andrew/Downloads/RPI-CNC-V400.zip (on Linux) reports:

Archive:  /home/andrew/Downloads/RPI-CNC-V400.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
8006926336  Def64N 2359888958  71% 2017-01-22 12:23 929beaa2  RPI_CNC_V4.img
--------          -------  ---                            -------
8006926336         2359888958  71%                            1 file

(and unzip is able to extract the file correctly).

thejoshwolfe commented 7 years ago

Here's what I know:

compression method 9 is called DEFLATE64, which is invented by PKWARE, the inventor in the zip file format.
DEFLATE64 is a proprietary algorithm, which doesn't necessarily mean open source software can't implement it, but it does make open source software sad to implement it (the .mp3 format is/was this kind of situation fwiw.).
zlib does not support DELFATE64 despite how easy it would be for them to support it. https://zlib.net/zlib_faq.html#faq40 their cited reasons are its proprietary specification and the algorithm being generally ineffective.
yauzl relies on zlib built into the core of node.js for decompression, and yauzl doesn't have any native code dependencies (other than the node core). so no gyp or waf or anything when npm install yauzl.
implementing a decompression algorithm in javascript is the wrong tool for the job due to performance and lack of "basic" primitive features like 64bit integers.
In the ~2 years of yauzl's "mainstream" adoption, this is the first report I've gotten of a compression method other than the two common ones 0 and 8.

All these points are arguments against taking action on this issue and simply resolving it as wontfix. However, I'm willing to consider ways to improve yauzl's feature set and adoption, and if that means either adding a native code dependencies or implementing a decompression algorithm in javascript, I'll at least consider it a little.

What i'd like to do next is do an even more thorough investigation of which zipfile reading libraries support compression methods other than 0 and 8.

It may take a week or so before I have time to do this, so in the immediate future, the direct answer to your question is: No, there are no plans to support compression methods other than 0 and 8.

lurch commented 7 years ago

Ahhh, thanks for the detailed info. I agree that if yauzl relies on zlib, but zlib does not support DELFATE64, then this will be a rather thorny problem to fix. I'll keep monitoring this issue, but I won't be surprised if you close it ;-) In the meantime, I'll try to contact the creators of that zipfile, and suggest they use a different compression method if they want their downloads to be "Etcher friendly".

(cc @jviotti )

thejoshwolfe commented 7 years ago

I checked the source code for various open source zipfile reading utilities and libraries around the internet, and here's what I found regarding which compression methods are supported:

yauzl, for comparison, supports methods 0 and 8.
node-unzip supports methods 0 and 8.
node-unzipper supports methods 0 and 8.
python 3's zipfile module supports methods 0, 8, 12, and 14.
openjdk 6's java.util.ZipFile class supports methods 0 and 8
7-Zip: probably supports lots of compression methods probably including 9. Their source code is very difficult to read.
InfoZIP's unzip supports methods 0, 1, 2-5, 6, 8, 9, and 12. (source archive download: ftp://ftp.info-zip.org/pub/infozip/src/unzip60.tgz , extract.c line 1703).
WinRAR does not appear to be open source. The source I found for unrar did not have zipfile reading capabilities.
Windows Compressed Folder utility is not open source.
Mac Archive Utility is not open source, and in general is pretty hopeless when it comes to standard behavior.

So my summary is that if you (or your clients or whoever) want compression method 9 to work, you have to use InfoZIP's unzip utility, which is the one you originally mentioned in this issue, or possibly 7-Zip.

Another possibility is to somehow use PKWARE's proprietary zipfile software, the people who originally made the zipfile format and specification, but I've never heard of anyone actually using their software, so I don't think this is a realistic option.

lurch commented 7 years ago

Wow, thanks for the detailed survey @thejoshwolfe ! :100:

Can anyone link me to any source file in 7-Zip's source code or to a direct download link for an archive file?

I think that's what http://7-zip.org/sdk.html is (at the time of writing, both that page and http://7-zip.org/download.html report the latest version as 17.00 beta with a date of 2017-04-29)

thejoshwolfe commented 7 years ago

Thanks for the tip on 7zip's source code. I found it, finally, and it's a real pain to read. I can't find their list of supported compression methods when reading .zip files, but it actually looks pretty thorough. I expect they do support compression method 9. I've updated my list above with this information.

lurch commented 7 years ago

Yeah, the 7z command line utility on Ubuntu 14.04 is able to decompress Deflate64 zip archives. It's also able to decompress LZMA and PPMd zip files, which InfoZIP's unzip isn't able to.

I had a rummage around in the 7zip source code and found this in DOC/Methods.txt (although of course just because an ID is listed it doesn't necessarily mean that 7zip supports it!)

   01 - [Zip]
      00 - Copy (not used. Use {00} instead)
      01 - Shrink
      06 - Implode
      08 - Deflate
      09 - Deflate64
      0A - Imploding
      0C - BZip2 (not used. Use {040202} instead)
      0E - LZMA (LZMA-zip)
      5F - xz
      60 - Jpeg
      61 - WavPack
      62 - PPMd (PPMd-zip)
      63 - wzAES

The hexadecimal codes in the table above match up with the decimal codes at http://www.winzip.com/comp_info.htm

alexwlchan commented 4 years ago

Thanks for the detailed info, this was super useful and saved me a bunch of research!

7-Zip: probably supports lots of compression methods probably including 9.

FWIW, I've been having this issue with an archive created with 7-Zip (I'm not sure which version, but probably new-ish). It does seem to support compression method 9. 😭

thejoshwolfe / yauzl

unsupported compression method: 9 #58