Compare the contents of the archive, not the archive binary itself

[sudden6]

This line https://github.com/NicoHood/gpgithub/blob/master/gpgtag.sh#L59 will probably fail, because of different compression settings on GH and locally.

To verify the the archive the following steps are necessary:

• extract the GH archive to DIR1
• extract our own archive to DIR2
• use diff to compare DIR1 and DIR2

[NicoHood]

Did it fail for you? I think github uses the same mechanism to generate their archives. This always worked for me, so unless someone proofs it wrong we can still use it I think.

Thanks for looking through the code.

[sudden6]

For me the hashes of the GH archive and my generated archive don't match, even though the contents are the same (verified with diff).

The point is, that the archive binary depends on more than the content, e.g. compression level, order of the files,... which makes your method unreliable at best.

[NicoHood]

Which file? So I can test it myself? The zip or the tar.gz or both?

[sudden6]

I only tested it with the tar.gz,

[NicoHood]

I am thinking of handling the process a bit different now:

The script is meant for super easy creation of signed releases, especially for github folks. It mainly wraps very simple gpg commands so if I make it more flexible you just can use the gpg commands on your own. Also the more complex it gets, the more users will refuse to use it and the more bugs it may have.

The idea is to create a straight forward script to create signed github releases. And the readme will explain what the script does for you, so if you ever need to do things on your own, you know how to do. But in the first place, the script will guide you.

The steps will be:

• Create new tag (if not existant)
• Generate new .tar.xz -9
• Sign the archive
• Upload it

Note that is does not verify the github source. The reason why I dont want to use it is because it requires a lot of coding, extracting, downloading etc and also this is more github specific. If i create a tar.xz we can use best compression and the only github dependent part is the uploading (which I still have not figured out how to do it super fancy).

Because otherwise you end up in adding thousands of params to download the correct file from github. This way you can just run the script and follow the instructions. KISS

[sudden6]

That would probably be the easier way.

[farseerfc]

Comes from https://github.com/qTox/qTox/issues/4045 A quick question, why not relay on the source code tarballs created by GitHub? By this line we have already created a release, all we need to do is to wait a few seconds, download the tarballs using the URL in this line , sign the tarball, and upload the signature. We can do this for all the tarballs and zips created by GitHub. Creating ${TAG}.tar.xz locally don't buy us any good in my opinion.

[NicoHood]

@farseerfc The problem is, that we cannot simply download the files without verifying the content. We would be fools if we do so. Because then we would not need GPG at all, as we simply trust the github https download.

So in order to verify we need to create our own .tar file and download the github tar. And (as said above) we need to extract and compare them because of different compression (otherwise cmp would be fine). Do not use pkgdiff, as it compares only the filenames, not the content. we need to use diff here.

So in the end you have 2 files: You tar and the github tar. And you also need to extract them. This gives you:

• A messy directory
• A lot of code to maintain/write
• Worse compression
• Github dependant code

Thatswhy I chose the .tar.xz creation for now. Feel free to implement a verification.

I will push a lot more commits in a few minutes, so you can review them if you wish. I am really happy with the result. I will possibly rename the script to gpgit, as it should not keep people with normal git repos from using gpg. I will add github features, such as uploading as extra option. Let me push the changes real quick.

[zetok]

Regarding xz usage: http://www.nongnu.org/lzip/xz_inadequate.html

[zetok]

Simple benchmark, result sorted according to the size in increasing order:

Size in bytes	archive	command
2582579	v1.7.1.tar.lz	lzip --best
2589828	v1.7.1.tar.xz	xz -9e
2630680	v1.7.1.tar.xz	xz --best

[NicoHood]

@zetok would you like to have a switch that uses different compression algorithms such as xz, gz, lz, bz2, zip etc?

If someone has a reasonable idea on how to implement the github diff comparison we could also "just" sign their release tarball. But we need to verify it. And we also need options for users without github.

[zetok]

@zetok would you like to have a switch that uses different compression algorithms such as xz, gz, lz, bz2, zip etc?

Why bother though, given that lzip seems like the best choice?

With https://github.com/qTox/qTox/pull/4047 the only missing part is a small program to upload stuff to a GH release.

The script above is generic, so it can be used for any git repo that has tags.

[sudden6]

https://github.com/TokTok/c-toxcore/pull/425 might be worth a look, it compares the content by overwriting the repo and using git diff.

[NicoHood]

@sudden6 Your script does not cover if the online resource has some file deleted. It is not correct like this.

BTW I just released a new version which enables all those new features. I am currently working on the github upload (which works, but is not error proof).

[sudden6]

Ok, didn't look close enough then.

[NicoHood]

Uploading to Github is now fully functional with error detection. I will release the next version soon. Feel free to correct basic bash issues, as I am no bash pro. Also feel free to correct the readme wording.

[NicoHood]

@sudden6 We are also building a wiki with a huge database about linux software security now. If you are interested in helping or just giving feedback, feel free to do so: https://github.com/NicoHood/gpgit/wiki

[zetok]

wiki is locked down

[NicoHood]

@zetok the wiki is only for people that i invite so nobody can change it without permission. If you wish to participate i can invite you. However I'd like to give you a short overview of the current plan:

As you can see we have a list of the software sources as well as a list for each distribution (currently only arch). The distribution lists will be generated via a script that parses the PKGBUILDs. So no need to edit anything here, I will do this the next weeks once the script is ready.

You can however add stuff to the software sources list. I will add more detailed guidelines soon. It would be nice if you can conform to the current layout/template. You might want to wait if you want to help adding more than a few packages. Otherwise you can also post them in the comments here (but please as source, so i can copy the markdown).

[zetok]

@zetok the wiki is only for people that i invite so nobody can change it without permission.

You're shooting yourself in the foot.

[NicoHood]

@zetok I am currently working on this in the background. I will automate the process and release it as separate project. This will take some more month of work though. Afterwards everyone can contribute. Thanks for your feedback.