Request to add uploaded source tarball to releases

[rfht]

Godot version: 3.0.2

OS/device including version: OpenBSD 6.3 -current

Issue description:

Hi,

We have a port of Godot for OpenBSD essentially ready, but there's the issue that there is no stable release tarball of the source available for download. The auto-generated tarballs have the unfortunate characteristic that their contents can change at the drop of a hat from github's side, e.g. when github updates its software and caches expire.

This is usually not an issue for individual downloads, but in our ports architecture, the distfiles are checked for size and hash and those can easily change and break the port if only the auto-generated tarballs are available.

The issue has been described in more detail on our mailing list. An example of a stable source tarball is here: https://github.com/rdoeffinger/iec16022/releases/ where the second download is the tarball of the distfiles.

I'd be happy to try in any way I can with this if there are questions. This is a cross-OS issue and as far as I know, other platforms with godot ports would also benefit.

Steps to reproduce: does not apply

Minimal reproduction project: does not apply

[akien-mga]

Thanks, I did not know that GitHub-generated tarballs were non deterministic, it can actually become an issue for our Asset Library which relies on the SHA-256 sums of such tarballs to validate downloads (cc @bojidar-bg).

Do you have more details on how those tarballs can end up differing from what git archive $tag --prefix=$name-$tag/ -o $name-$tag.tar.gz would generate locally?

At any rate, I've uploaded source tarballs for 3.0, 3.0.1 and 3.0.2 on our downloads repository: https://downloads.tuxfamily.org/godotengine/3.0.2/godot-3.0.2-stable.tar.xz (SHA-256 sum: https://downloads.tuxfamily.org/godotengine/3.0.2/godot-3.0.2-stable.tar.xz.sha256).

[rfht]

I asked on our porters chat for some examples that I could provide to you. Here's what I got:

git tarballs are generated with git-archive and then cached. so if they're generated on a node with one set of software versions, the bitstream may vary from a file generated on another node with different versions.
one concrete example: in the past they had a patched version of tar to cope with an issue in openbsd tar, which has long since been fixed on our side. sometime (last year or so?) they dropped that, so files generated after that point had a changed tar bitstream
other changes to things like git, zlib, tar might cause other differences in future

https://www.conserver.com/pipermail/users/2018-March/msg00006.html

https://github.com/pgRouting/pgrouting/issues/1013

Hope this helps you understand the issue for your needs as well!

[bojidar-bg]

@akien-mga we still haven't observed a single source .zip which has changed, but since we are going to support GH releases soon, I guess it would stop being that much of a problem.

[rfht]

@akien-mga hi, are you planning to upload source tarballs for 3.0.3 to https://downloads.tuxfamily.org/godotengine/3.0.3/ like you did with 3.0.2? Thanks!

[akien-mga]

@rfht Done: https://downloads.tuxfamily.org/godotengine/3.0.3/godot-3.0.3-stable.tar.xz

@hpvb Something to add to your build script (though, it can only be done once 3.0.3-stable is tagged):

git archive ${VERSION}-stable --prefix=godot-${VERSION}-stable/ -o ../godot-${VERSION}-stable.tar.xz
cd ..
sha256 godot-${VERSION}-stable.tar.xz > godot-${VERSION}-stable.tar.xz.sha256
// upload to SSH

It's probably not necessary for pre-releases (betas, RCs, etc.).

[rfht]

Thanks so much! 3.0.3 works flawlessly on light testing! From my side, I'm planning to only follow the releases with the OpenBSD port at this point.