Possibility of adding release tarball generation

[qbit]

In the OpenBSD ports tree we have encountered numerous issues with GitHub generated tarballs. Would it be possible to get manually generated tarballs attached as a release?

Here is some backstory on the push for changes in the OpenBSD ports tree. Please note that this has impact outside of OpenBSD.

[stealth]

OK, so what would be your proposal? Theres no problem on my side mirroring explicit tarballs on some OpenBSD infra. Otherwise I would have to set up something just to provide these tarballs?

[qbit]

Most projects seem to be opting for attaching to tags, it's free and still within github.

[stealth]

You must be trolling me. opmsg has 23 releases so far, all properly tagged and even signed. So it should be possible to get the release tarballs via gh?

[qbit]

The git commits are signed, yes. But the tarballs currently available via github can change at any point, which breaks port builds during the 6 month life span of an OpenBSD release.

OpenBSD doesn't have (and likely won't any time soon) a way to download ports via git directly. This is because in-base tools like ftp(1) are preferred as they use more of OpenBSD's security measures (like pledge).

I can create and host tarballs myself, however, this only helps OpenBSD. Other projects that use github's tarballs will suffer the same issues.

[sthen]

Hi, another OpenBSD ports dev here who has been running into this recently. Signing commits is great but only relates to the files stored in git. I asked a github developer about this as we've seen files with changing hashes over the last few months. The downloadable source code zip and tar.gz files on github are generated with git-archive then cached, there isn't anything to ensure a consistent file which can vary depending on versions of various software - git, tar, zip/libz. He included this in his reply:

These are not planned changes but rather they come about from updating
the software involved in creating them. The main purpose of the auto-
generated archives are for someone to download the source from the
website if they don't want to bother with downloading the repository.

It is not meant to be reliable or a way to distribute software releases
and nothing in the software stack is made to try to produce consistent
archives. This is no different from creating a tarball locally and
trying verify it with the hash of the tarball someone created on their
own machine.

The only way to get a known-good checksum for a tarball is to have
upstream (or the packagers) prepare the release and upload the tarball
alongside its checksum. This is true regardless of GitHub. There is a
feature on the site where maintainers can upload their own assets for a
release though clearly not too many people actually use it.

It's a bit annoying that an extra step is involved to generate and upload a tar but it's quite widely done (irssi, mame, libgd, darktable, roundcube, libevent, protobuf, transmission, ...) because it means that as project developer you're in control of the exact file people are downloading (and can sign it so users can be sure it wasn't tampered with).

[stealth]

Hm, OK. I dont have any https:// site, so I could upload tarballs at http://stealth.openwall.net/crypto/ The question is where to put the sha256sums

[sthen]

No need to self-host for these, github will host them for you. Step 7 on https://help.github.com/articles/creating-releases/ shows how to do this manually, or if you'd like to script it there are examples at https://gist.github.com/stefanbuck/ce788fee19ab6eb0b4447a85fc99f447 and https://stackoverflow.com/questions/40733692/github-upload-release-assets-with-bash

[stealth]

Ok, if its just that. There should be a 1.77s release now...

[stealth]

Hm, step 7... What a mess. You mean I shall setup an extra repo to include my own tarballs as a binary blob into a release, or upload a tarball into the opmsg release? Thats really strange idea. That would mean the download archive contains another opmsg tarball

[sthen]

They don't go into a git repository at all, they are held separately but associated with the release. Releases -> create a new release -> add tar to the "Attach binaries by dropping them here or selecting them" box. It looks like it should be quite simple?

[stealth]

Its easy enough for me, but really looks like a very dirty workaround for a github problem. I consider to come up with something better. What about distributing official tarballs externally plus .opmsg signature files, so you can verify it with an existing opmsg install? Is that possible for ports, or is there a bootstrapping problem on your side? Optionally, also providing gpg signatures for the tarballs. If thats not an option for you, I may create a distro repo here, to just upload tarballs, which should avoid the on-the-fly-archive problem.

[qbit]

We can grab the tarballs from any URL with equal ease. The port system doesn't have a way to verify external signatures (gpg, opmsg), but typically if sigs are provided I will verify them when an upstream has a new release. I have opmsg and gpg so I can verify sigs done with either.

[stealth]

Ok, gentlemen. Please check the release subdir of my new push. That should address it.

[qbit]

Looks good! Thanks!