Comparing local-generated archive vs github-generated archive fails

[cmaglie]

For some reason the tar generated locally differs from the one generated from github

    Verifying git against local source.
+ gpgit_yesno
+ [[ false == true ]]
+ read -r -t 0
+ read -rp '    Continue? [Y/n]' yesno
    Continue? [Y/n]
+ [[ '' != [Yy]\e\s ]]
+ [[ '' != [Yy] ]]
+ [[ -n '' ]]
+ git archive --format=tar --prefix Arduino-1.8.2/ 1.8.2
+ cmp /dev/fd/63
++ gzip -dc /home/megabug/Code-NOBCK/Arduino/archive/Arduino-1.8.2.tar.gz
/dev/fd/63 - differ: char 37358081, line 478810
+ error 'Archive differs from local source.'
+ local 'mesg=Archive differs from local source.'
+ shift
+ printf '==> ERROR: Archive differs from local source.\n'
==> ERROR: Archive differs from local source.
+ exit 1

It may be that comparing byte-by-byte is too much? Probably different version of tar may produce different binary archives.

Also I'm wondering if this check is worth at all since we are always going to upload our own produced source archives.

[NicoHood]

This check is required if you want to use the github generated .tar.gz. With your own upload this is of course not required.

Could you please give me sha512sums of the Github .tar.gz, .tar and the local .tar.gz and .tar? This way I can compare what my Archlinux tar gives as output.

[cmaglie]

$ wget https://github.com/arduino/Arduino/archive/1.8.2.tar.gz
--2017-03-30 17:03:34--  https://github.com/arduino/Arduino/archive/1.8.2.tar.gz
Risoluzione di github.com (github.com)... 192.30.253.112, 192.30.253.113
Connessione a github.com (github.com)|192.30.253.112|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 302 Found
Posizione: https://codeload.github.com/arduino/Arduino/tar.gz/1.8.2 [segue]
--2017-03-30 17:03:35--  https://codeload.github.com/arduino/Arduino/tar.gz/1.8.2
Risoluzione di codeload.github.com (codeload.github.com)... 192.30.253.121, 192.30.253.120
Connessione a codeload.github.com (codeload.github.com)|192.30.253.121|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 200 OK
Lunghezza: 48191808 (46M) [application/x-gzip]
Salvataggio in: "1.8.2.tar.gz"

1.8.2.tar.gz                                  100%[==========================================================>]  45,96M  1,27MB/s    in 60s     

2017-03-30 17:04:36 (778 KB/s) - "1.8.2.tar.gz" salvato [48191808/48191808]
$ mv 1.8.2.tar.gz Arduino-1.8.2-github.tar.gz
$ gunzip Arduino-1.8.2-github.tar.gz 
$ git archive --format=tar --prefix Arduino-1.8.2 1.8.2 > Arduino-1.8.2-local.tar
$ ls -ln Arduino*tar
-rw-rw-r-- 1 1000 1000 91678720 mar 30 17:04 Arduino-1.8.2-github.tar
-rw-rw-r-- 1 1000 1000 91678720 mar 30 17:05 Arduino-1.8.2-local.tar
$ sha512sum Arduino*.tar
19986c2376665f82af582a3be5f687e65ac5fccaba3147844b591261b0d51fe45f54ae497d1dd1e79ca7f1385b9567bdab1b0cfed33ee1ee8e245e9ac964fff2  Arduino-1.8.2-github.tar
c2345abef64d2b11bbbdcef6e0523ec8d76dca293eaab87865e5f64f50b5a4dd6f70398af6fb6b16a5ac1e41ef9d4238496b33fe9244446480f5d0ade7e7b4a8  Arduino-1.8.2-local.tar

[cmaglie]

Ops forget again the / btw the result is still different from github:

$ git archive --format=tar --prefix Arduino-1.8.2/ 1.8.2 > Arduino-1.8.2-local.tar
$ ls -ln Arduino*tar
-rw-rw-r-- 1 1000 1000 91678720 mar 30 17:04 Arduino-1.8.2-github.tar
-rw-rw-r-- 1 1000 1000 91678720 mar 30 17:08 Arduino-1.8.2-local.tar
$ sha512sum Arduino*.tar
19986c2376665f82af582a3be5f687e65ac5fccaba3147844b591261b0d51fe45f54ae497d1dd1e79ca7f1385b9567bdab1b0cfed33ee1ee8e245e9ac964fff2  Arduino-1.8.2-github.tar
6e61ad44ac2718534ec104368ed1315b15166facb7309ac26df3a8ccbab58d0691cca1d4603c6280ebe1ec3218070fe8d7fcc82e2ad18a587e3d57db50375645  Arduino-1.8.2-local.tar

[cmaglie]

Also this may help:

$ cmp Arduino-1.8.2-github.tar Arduino-1.8.2-local.tar 
Arduino-1.8.2-github.tar Arduino-1.8.2-local.tar differenza: byte 37358081, riga 478810

BTW I don't know if it's worth pursuing this thing, IMHO you can safely remove this feature and greatly simplify your script.

[NicoHood]

@cmaglie Thanks for the report, I will have a detailed look.

BTW I don't know if it's worth pursuing this thing, IMHO you can safely remove this feature and greatly simplify your script.

The sense of gpgit is to sign the sourcecode that you trust. So if you just download it from github and sign it, you trust github to create it properly, their servers and the https CA and your firewall. You just sign the public available file. But I can also download this file, so its more or less just a hash comparison between your and my download. But it does not ensure that the code is exactly what you have locally. And thatswhy it is important.

If this all does not work I need to find another solution or release files in .tar.gz need to be duplicated (but .xz is still better, so might be less of a problem).

[cmaglie]

Consider also that github generates the downloads and, AFAIK, there are no guarantees that those downloads are stable over time, I think that they are subject to change when the gzip or tar or any other tools is updated on github's servers: I remember that this already happened, and caused a small denial-of-service, with the library manager (when we used to link directly github instead of hosting the libraries archives by ourself).

[NicoHood]

Will be dropped in the next version of GPGit (python). Thanks for the advice.

[NicoHood]

gpgit will always generate tarballs from the local git now.