Problems to unzip file / support zip64 locator format

[beckmi]

I have a zip archive with only one compressed file in it. Using unzip it is possible to uncompress it. Using zziplib it is not possible because the dirent of this file is as follows:

(gdb) p dirent $4 = {z_magic = "PK\001\002", z_encoder = {version = "-", ostype = ""}, z_extract = {version = "-", ostype = ""}, z_flags = "\b", z_compr = "\b", z_dostime = {time = "\353S", date = "GQ"}, z_crc32 = "|#\301\215", z_csize = "\377\377\377\377", z_usize = "\377\377\377\377", z_namlen = "\034", z_extras = "\034", z_comment = "\000", z_diskstart = "\000", z_filetype = "\000", z_filemode = "\000\000\000", z_offset = "\377\377\377\377"}

It is possible to list the content, but it is not possible to open the compressed file, because z_offset is -1. Attached the file in question. It is from the sftp side of ENTSOE-E. All their compressed files have the same problem.

[beckmi]

2020_10_OutagesPUReasons.zip

the lost attachment ...

[gdraheim]

It seems that PKWARE has changed its published standards.

4.3.9.2 When compressing files, compressed and uncompressed sizes SHOULD be stored in ZIP64 format (as 8 byte values) when a file's size exceeds 0xFFFFFFFF. However ZIP64 format MAY be used regardless of the size of a file. When extracting, if the zip64 extended information extra field is present for the file the compressed and uncompressed sizes will be 8 byte values.

4.4.16 relative offset of local header: (4 bytes) This is the offset from the start of the first disk on which this file appears, to where the local header SHOULD be found. If an archive is in ZIP64 format and the value in this field is 0xFFFFFFFF, the size will be in the corresponding 8 byte zip64 extended information extra field.

4.5.3 -Zip64 Extended Information Extra Field (0x0001): The following is the layout of the zip64 extended information "extra" block. If one of the size or offset fields in the Local or Central directory record is too small to hold the required data, a Zip64 extended information record is created. The order of the fields in the zip64 extended information record is fixed, but the fields MUST only appear if the corresponding Local or Central directory record field is set to 0xFFFF or 0xFFFFFFFF.

Note: all fields stored in Intel low-byte/high-byte order.

 Value                 Size       Description

0x0001 2 bytes Tag for this "extra" block type (ZIP64) Size 2 bytes Size of this "extra" block Original Size 8 bytes Original uncompressed file size Compressed Size 8 bytes Size of compressed data Relative Header Offset 8 bytes Offset of local header record Disk Start Number 4 bytes Number of the disk on which this file starts

So far the zziplib can read a ZIP64 central directory but it does not read a ZIP64 extras block.

The real bug here is the fact that the file you provided does NOT provide a ZIP64 central directory (magic PK\6\6) but only a normal ZIP central directory (magic PK\6\5) so that the use of a ZIP64 extras block is atleast unintended .... as the usage of 0xFFFF as an extension marker was defined for the ZIP64 file format.

It could be implemented however.

[gdraheim]

After a bit more debugging I can see that the ZIP64-trailer is not used but instead there is a ZIP64-locator (PK\6\7). The pkware appnote documentation says that it was introduced in version 6.2 in 2004/2005. Bewildering as it may seem but the zziplib is older (going back to the 1990ies).

I did check if I can implement the functionality but quite some logic needs to be changed here, so it is a real feature request instead of just a bug fix. I am sorry but this will not come around anytime soon.

[beckmi]

Nevertheless – thanks for the work done so far. Is an open spec downloadable?

Have found it. For others who search it: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT

[gdraheim]

Yes, that's it - when pkware did create the first zip.exe they were shipping the package with a APPNOTE.TXT file which did describe (parts of) the file format. That name has stuck referring to the standardisation proposal later.

Here's the official archive = https://support.pkware.com/home/pkzip/developer-tools/appnote/application-note-archives