search

nih-at / libzip

A C library for reading, creating, and modifying zip archives.
https://libzip.org/
Other
836 stars 272 forks source link

libzip/build/src/zipcmp: cannot open zip archive 'orig.zip': Zip archive inconsistent #232

Closed stdedos closed 3 years ago

stdedos commented 3 years ago

Describe the Bug I am trying to use zipcmp to compare two otherwise similar zip files:

$ colordiff -su <(unzip -l orig.zip | awk '{print $4}' | sort) <(unzip -l new-0.3.0.zip | awk '{print $4}' | sort)
--- /proc/self/fd/11    2021-02-24 20:54:12.277340132 +0200
+++ /proc/self/fd/12    2021-02-24 20:54:12.285340296 +0200
@@ -1,6 +1,7 @@

+
 ----
 Name
 new/

Both zipcmp (libzip 1.1.2) and 6ca670c8 zipcmp give:

libzip/build/src/zipcmp: cannot open zip archive 'orig.zip': Zip archive inconsistent

The error, while useful to some people:

[ZIP_ER_INCONS]    Inconsistencies were found in the file specified by path.  This error is often caused by specifying ZIP_CHECKCONS but can also happen without it.

does not really specify what is the problem.

Also, since the archive is openable, extractable, etc, I am not sure why can't zipcmp continue its task.

Also, I don't know what happens when the files are matching, neither when there are differences. Any information on stdout, including Files a and b match (akin to diff -s) would be welcome.

Operating System Ubuntu 18.04

0-wiz-0 commented 3 years ago

If you upload the zipfile, I can probably tell you more details. Other than that, I'm not sure what the actionable items here are.

stdedos commented 3 years ago

Oh, oops 😄 I should've started like that:

orig.zip new-0.3.0.zip

0-wiz-0 commented 3 years ago

new-0.3.0.zip is fine. orig.zip seems to have been made with a version of infozip that creates zip archives that do not conform to the zip spec in a minor detail.

We've changed the zipcmp -p flag to not check for inconsistencies by default. If you still want to do that, use -C -and it also provides a bit more detail now.

Check https://github.com/nih-at/libzip/commit/1bd794f357231f304fe67dbb76d8245e887e6160 or newer and let us know!

Thanks for the test files!

stdedos commented 3 years ago

Okay, so - I've tried it now:

$ libzip/build/src/zipcmp -pvt orig.zip new-0.3.0.zip

and I get:

--- orig.zip
+++ new-0.3.0.zip
+                     service.subtitles.bsplayer/  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/  5455 l <07 a4 91 35 60 a5 91 35 60 a4 91 35 60>
-                     service.subtitles.bsplayer/  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/  5455 c <07 a4 91 35 60 a5 91 35 60 a4 91 35 60>
-                     service.subtitles.bsplayer/  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/LICENSE  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/LICENSE  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/LICENSE  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/LICENSE  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/LICENSE  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/LICENSE  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/README.md  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/README.md  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/README.md  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/README.md  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/README.md  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/README.md  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/addon.xml  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/addon.xml  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/addon.xml  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/addon.xml  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/addon.xml  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/addon.xml  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/changelog.txt  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/changelog.txt  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/changelog.txt  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/changelog.txt  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/changelog.txt  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/changelog.txt  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/icon.png  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/icon.png  5455 l <07 b9 7b 35 60 bd 7d 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/icon.png  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/icon.png  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/icon.png  5455 c <07 b9 7b 35 60 bd 7d 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/icon.png  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/logo.png  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/logo.png  5455 l <07 b9 7b 35 60 bd 7d 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/logo.png  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/logo.png  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/logo.png  5455 c <07 b9 7b 35 60 bd 7d 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/logo.png  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/__init__.py  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/__init__.py  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/__init__.py  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/__init__.py  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/__init__.py  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/__init__.py  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
---                     service.subtitles.bsplayer/resources/__init__.py  method 8
+++                     service.subtitles.bsplayer/resources/__init__.py  method 0
+                     service.subtitles.bsplayer/resources/language/  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/language/  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/language/  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/language/  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/language/  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/language/  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/language/English/  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/language/English/  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/language/English/  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/language/English/  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/language/English/  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/language/English/  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/language/English/strings.po  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/language/English/strings.po  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/language/English/strings.po  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/language/English/strings.po  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/language/English/strings.po  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/language/English/strings.po  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/lib/  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/lib/  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/__init__.py  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/__init__.py  5455 l <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/lib/__init__.py  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/__init__.py  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/__init__.py  5455 c <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-                     service.subtitles.bsplayer/resources/lib/__init__.py  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
---                     service.subtitles.bsplayer/resources/lib/__init__.py  method 8
+++                     service.subtitles.bsplayer/resources/lib/__init__.py  method 0
+                     service.subtitles.bsplayer/resources/lib/bsplayer.py  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/bsplayer.py  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/lib/bsplayer.py  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/bsplayer.py  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/bsplayer.py  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/lib/bsplayer.py  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/utils.py  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/utils.py  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/lib/utils.py  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/resources/lib/utils.py  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/resources/lib/utils.py  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/resources/lib/utils.py  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/service.py  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/service.py  5455 l <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/service.py  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>
+                     service.subtitles.bsplayer/service.py  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/service.py  5455 c <07 23 85 35 60 23 85 35 60 23 85 35 60>
-                     service.subtitles.bsplayer/service.py  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>
--- archive comment (0)
+++ archive comment (40)
  1. Pardon my ignorance, but ... what am I seeing? 😕 Archives should have the same files "more or less", with the exception of contents. However, almost for each file I have two deletions and one addition! Wat?? Also some explanation on what am I seeing would be nice; I don't see anything e.g. on the man page
  2. Pardon my pedanticism, but ... usually, from diff point of view, first you remove lines, and then you add them.
  3. Why do empty files get a special treatment, and get marked with ---, +++? 
    ---                     service.subtitles.bsplayer/resources/__init__.py  method 8
+++                     service.subtitles.bsplayer/resources/__init__.py  method 0
  4. --- archive comment (0)
+++ archive comment (40)

    Kewl, but ... can I see what its diff is? 😝

0-wiz-0 commented 3 years ago

Excuse the rough edges - you're probably the first person except the developers that actually used this :) It's mostly used for automated testing, where we don't want to find any differences. As for your questions:

  1. Well the files do diff in the extra fields a lot. Most of the entries you see are of the form: 
    filename extrafield-id local/central details

    so for example

    +                     service.subtitles.bsplayer/  5455 l <01 8b 84 35 60>
-                     service.subtitles.bsplayer/  5455 l <07 a4 91 35 60 a5 91 35 60 a4 91 35 60>
-                     service.subtitles.bsplayer/  7875 l <01 04 e8 03 00 00 04 e8 03 00 00>

    means that orig.zip has two extra fields in the local header, one for id 5455 with the bytes <07 a4...> as contents and one for id 7875 with <01 04...>, while new-0.3.0.zip only has one extra field for 5455 with fewer bytes instead. This block

    +                     service.subtitles.bsplayer/  5455 c <01 8b 84 35 60>
-                     service.subtitles.bsplayer/  5455 c <07 a4 91 35 60 a5 91 35 60 a4 91 35 60>
-                     service.subtitles.bsplayer/  7875 c <01 04 e8 03 00 00 04 e8 03 00 00>

    compares the extra field in the central directory and finds the same differences there. I think this covers >90% of the diffs.

  2. It doesn't really matter, does it? :)
  3. it's not special about empty files, it's just that the file compression method diffs here between the files. One zip has method 8 (deflate) while the other has method 0 (stored). There's not much point setting the method to deflate for a 0-byte file, but on the other hand it doesn't really matter.
  4. Archive comments can be quite huge. You can output the archive comment with ziptool filename get_archive_comment.
stdedos commented 3 years ago

For [1], I see what you are saying, but I still don't understand what I am seeing. A lot of files have a lot of common headers, even though they are different files 😕

I was looking for a program that would compare two zip archive content properties (filenames, sizes, hashes), and even maybe "advanced stuff" (extra fields etc) without having to decompress them. I guess I found a different program instead. 😃

Thank you for your time and patience 😀

0-wiz-0 commented 3 years ago

We've improved the output of zipcmp, give it another try!

stdedos commented 3 years ago

With:

--- service.subtitles.bsplayer.zip
+++ service.subtitles.bsplayer-0.3.0.zip
  file 'service.subtitles.bsplayer/', size 0, crc 00000000
+   local extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   local extra field extended timestamp, length 13, data <07 a4 91 35 60 a5 91 35 60 a4 91 35 60>
-   local extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
+   central extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   central extra field extended timestamp, length 13, data <07 a4 91 35 60 a5 91 35 60 a4 91 35 60>
-   central extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
  file 'service.subtitles.bsplayer/LICENSE', size 18047, crc a36db529
+   local extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   local extra field extended timestamp, length 13, data <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-   local extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
+   central extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   central extra field extended timestamp, length 13, data <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-   central extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
...
  file 'service.subtitles.bsplayer/resources/__init__.py', size 47, crc 0f5b458a
-   compression method Deflated
+   compression method Stored (no compression)
+   local extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   local extra field extended timestamp, length 13, data <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-   local extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
+   central extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   central extra field extended timestamp, length 13, data <07 b9 7b 35 60 ca 7b 35 60 b9 7b 35 60>
-   central extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
...
-   central extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
  file 'service.subtitles.bsplayer/service.py', size 3299, crc 68368765
+   local extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   local extra field extended timestamp, length 13, data <07 23 85 35 60 23 85 35 60 23 85 35 60>
-   local extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
+   central extra field extended timestamp, length 5, data <01 8b 84 35 60>
-   central extra field extended timestamp, length 13, data <07 23 85 35 60 23 85 35 60 23 85 35 60>
-   central extra field Info-ZIP UNIX 3rd generation, length 11, data <01 04 e8 03 00 00 04 e8 03 00 00>
+ archive comment, length 40, data <33 34 34 34 36 33 36 39 30 30 32 33 66 38 34 64 36 35 35 30 34 62 31 31 34 35 30 36 35 39 63 63 33 38 39 33 64 65 61 39>

However, given that this is an internal tool, as you said - keep these for your reference only.

I have compared similar archives with same files but different contents, and the format appears to be very nice. All in all, I am satisfied to use this program for "basic" comparison of a zip archive, even as a basic user.