zip64 extension does not respect versionMadeBy/versionNeededToExtract settings at member level

farblos commented 5 years ago

The zip64 PR (my own, unfortunately) merged into version 1.66 uses hard-coded values to set attributes versionMadeBy and versionNeededToExtract when writing local and CD headers. It does not use the values available in the Archive::Zip::Member object.

I noticed that when trying to establish round-trip-equality, that is, read some arbitrary zip file with Archive::Zip, write it out again, and compare the result to the original one.

Not sure what an "ideal" solution to this problem would be.

farblos commented 5 years ago

The following logic seems reasonable:

for versionMadeBy, use default value 20 (45 for zip64 EOCD record) or any previously read value. Never change that value when writing.
likewise for versionNeededToExtract, but force a minimum value of 45 when writing headers in zip64 format or the zip64 EOCD record

Currently testing that logic ...

redhotpenguin commented 5 years ago

Great, send a PR when you're ready.

On Wed, Sep 18, 2019 at 7:16 AM farblos notifications@github.com wrote:

The following logic seems reasonable:

-

for versionMadeBy, use default value 20 (45 for zip64 EOCD record) or any previously read value. Never change that value when writing.

likewise for versionNeededToExtract, but force a minimum value of 45 when writing headers in zip64 format or the zip64 EOCD record

Currently testing that logic ...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/redhotpenguin/perl-Archive-Zip/issues/51?email_source=notifications&email_token=AAAXRZPXG32EWYVNZPDSCQLQKIZ2TA5CNFSM4IXSQDY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7AG4WY#issuecomment-532704859, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAXRZMJIY22BRQYZ4OEKG3QKIZ2TANCNFSM4IXSQDYQ .

pmqs commented 5 years ago

I checked a zip64 archive created with InfoZip, to see what it does.

First create a zip64 archive (the -fz option will force Zip64)

# zip -fz /tmp/test-deflate.zip ~/.bashrc 
updating: root/.bashrc (deflated 29%)

Then dump the details (see below). The Extract Zip Spec is 2D (i.e. 45 decimal) in both the local and central headers, so that matches with what you arer proposing. The Created Zip Spec is 1E (i.e. decimal 30) in the central headers. That doesn't match what you are proposing. Can't find a definition of the 3.0 version in the PKWARE APPNOTE.

Interestingly, if I use InfoZip to create a non-Zip64 archive, it does use 14 (i.e. 20 decimal).

No idea what the rationale is behind that is.

# zipdetails /tmp/test-deflate.zip

0000 LOCAL HEADER #1       04034B50
0004 Extract Zip Spec      2D '4.5'
0005 Extract OS            00 'MS-DOS'
0006 General Purpose Flag  0000
     [Bits 1-2]            0 'Normal Compression'
0008 Compression Method    0008 'Deflated'
000A Last Mod Time         439D1350 'Sun Dec 29 02:26:32 2013'
000E CRC                   ADF27BAA
0012 Compressed Length     FFFFFFFF
0016 Uncompressed Length   FFFFFFFF
001A Filename Length       000C
001C Extra Length          0030
001E Filename              'root/.bashrc'
002A Extra ID #0001        5455 'UT: Extended Timestamp'
002C   Length              0009
002E   Flags               '03 mod access'
002F   Mod Time            52BF8857 'Sun Dec 29 02:26:31 2013'
0033   Access Time         5D822A9A 'Wed Sep 18 14:01:14 2019'
0037 Extra ID #0002        7875 'ux: Unix Extra Type 3'
0039   Length              000B
003B   Version             01
003C   UID Size            04
003D   UID                 00000000
0041   GID Size            04
0042   GID                 00000000
0046 Extra ID #0003        0001 'ZIP64'
0048   Length              0010
004A   Uncompressed Size   00000000000000B0
0052   Compressed Size     000000000000007D
005A PAYLOAD               M.;..0.E.:..'Q....y.Q.(...a$.d..O.)hOqu;
                           ..kKe...&....W....4.4.o.W...........?p..
                           ...nM.....y.,..<..Y.&=...B.1........".n.
                           ..+}.

00D7 CENTRAL HEADER #1     02014B50
00DB Created Zip Spec      1E '3.0'
00DC Created OS            03 'Unix'
00DD Extract Zip Spec      2D '4.5'
00DE Extract OS            00 'MS-DOS'
00DF General Purpose Flag  0000
     [Bits 1-2]            0 'Normal Compression'
00E1 Compression Method    0008 'Deflated'
00E3 Last Mod Time         439D1350 'Sun Dec 29 02:26:32 2013'
00E7 CRC                   ADF27BAA
00EB Compressed Length     0000007D
00EF Uncompressed Length   FFFFFFFF
00F3 Filename Length       000C
00F5 Extra Length          0024
00F7 Comment Length        0000
00F9 Disk Start            0000
00FB Int File Attributes   0001
     [Bit 0]               1 Text Data
00FD Ext File Attributes   81A40000
0101 Local Header Offset   00000000
0105 Filename              'root/.bashrc'
0111 Extra ID #0001        5455 'UT: Extended Timestamp'
0113   Length              0005
0115   Flags               '03 mod access'
0116   Mod Time            52BF8857 'Sun Dec 29 02:26:31 2013'
011A Extra ID #0002        7875 'ux: Unix Extra Type 3'
011C   Length              000B
011E   Version             01
011F   UID Size            04
0120   UID                 00000000
0124   GID Size            04
0125   GID                 00000000
0129 Extra ID #0003        0001 'ZIP64'
012B   Length              0008
012D   Uncompressed Size   00000000000000B0

0135 ZIP64 END CENTRAL DIR 06064B50
     RECORD
0139 Size of record        000000000000002C
0141 Created Zip Spec      1E '3.0'
0142 Created OS            03 'Unix'
0143 Extract Zip Spec      2D '4.5'
0144 Extract OS            00 'MS-DOS'
0145 Number of this disk   00000000
0149 Central Dir Disk no   00000000
014D Entries in this disk  0000000000000001
0155 Total Entries         0000000000000001
015D Size of Central Dir   000000000000005E
0165 Offset to Central dir 00000000000000D7

016D ZIP64 END CENTRAL DIR 07064B50
     LOCATOR
0171 Central Dir Disk no   00000000
0175 Offset to Central dir 0000000000000135
017D Total no of Disks     00000001

0181 END CENTRAL HEADER    06054B50
0185 Number of this disk   0000
0187 Central Dir Disk no   0000
0189 Entries in this disk  0001
018B Total Entries         0001
018D Size of Central Dir   0000005E
0191 Offset to Central Dir FFFFFFFF
0195 Comment Length        0000

farblos commented 5 years ago

Thanks for the background infomation.

Since I couldn't make anything reasonable of the "2.0" and "3.0" for versionMadeBy/Created Zip Spec as well, I took them to be the version of InfoZip itself. It's only an educated guess, but at least it matches InfoZip's history: Large file support is available with InfoZip 3.0.

But anyway, I very much guess that no code actually cares about these values. My rationale for above logic was: Maintain round-trip equality where possible, but at least enforce the 45 for versionNeededToExtract/Extract Zip Spec in zip64 structures.

I added some short explanation on the logic to the POD as well (in "BUGS AND CAVEATS"), to have it at least documented.

lamby commented 5 years ago

Just an FYI we noticed this in Debian here: https://bugs.debian.org/940973#27

farblos commented 5 years ago

Issue fixed in version 1.67.

redhotpenguin / perl-Archive-Zip

zip64 extension does not respect versionMadeBy/versionNeededToExtract settings at member level #51

for versionMadeBy, use default value 20 (45 for zip64 EOCD record) or any previously read value. Never change that value when writing.