preparations for reproducible builds

[mark-kubacki]

This PR introduces requirements for reproducible builds except for changes to dgr itself.

For example, and updated tar is needed, and we could very well make more changes to the supplied rootfs.tar, so it needs to be easily comparable and reviewable.

[n0rad]

I don't see where you want to go by download and building your own tar. Current builder already have version 1.28. Is there a problem with this version ?

[mark-kubacki]

Yes, tar 1.29 is needed for the intended purpose. Please see 1daca96 for the usage example.

When you update the builder in the future, and it is on par with the "official" version, you'd just delete that file or git reverse.

[n0rad]

Ok, also, I'm thinking about building aci-builder & aci-tester with dgr. This would be cleaner

[mark-kubacki]

Yes, I've read about this while going through the issue tracker. You could merge this PR and we'd not need to wait for #140.

[mark-kubacki]

The way to see changes, as described in f2d792b, works as well for *.aci! (Though any compression cannot be detected.)

git config --global diff.aci.textconv "tar -tvf"
git config --global diff.aci.cachetextconv true

printf "\n*.aci           binary diff=aci\n" >>.gitattributes

This is one reason why it's important to sort contents of tarballs and archives.

[n0rad]

I'm more thinking about isolating all buildroot, tar, etc... stuff in a container and result with the aci directly.

[mark-kubacki]

An aci-builder, built by dgr itself, is at least a week away. That's my estimation.

I'd happily help you with that. An Ubuntu without its package manager is something that can easily be assembled, and with no more than systemd, libc, and tar, I estimate it will be ≤30 MiB before having been compressed. (See rkt fetch blitznote.com/ubuntu:16.10 and observe how big perl, apt, dpkg are.)

Anyway, that's out of scope of this PR. Given your plans we'd not need to reorder rootfs.tar and I can strip it from the PR. Would that move things forward for a tar v1.29? I'd like have this for later patches to dgr to use its options.

[mark-kubacki]

I've revised my patches based on the discussion. Given that the rootfs.tar.xz will be obsoleted I've dropped reordering or modifying it in any other way.

[n0rad]

The PR break the container build. I don't know where yet, but building a sample aci fail

[mark-kubacki]

Sorry. I will look into this in a few hours.

[n0rad]

At the same time, I think that updating tar in the builder this way is not good. Instead of downloading tar and building it on our own, it will be better to update build-root that probably have the latest. Moverover, instead of using tar to prepare, using a pure go version will be better too.

[mark-kubacki]

The version written in Go is currently not close to being on par with GNU tar.

Sorry to see the whole PR having been reverted. I would have loved to see an error message first.

[mark-kubacki]

Seems to me the new tar has not been included into the aci-builder, or the new aci-builder has not been used (yet): https://travis-ci.org/blablacar/dgr/builds/179487323#L592

Anyway, considering a hard fork.

[n0rad]

Latest buildroot include tar:1.29, we just have to update the buildroot and it should be good to go.

In the mean time, we have feature to release by tomorrow, this is why I revert the PR.

[mark-kubacki]

This would've been fine:

git cherry-pick 868a262 9ac59c0 bcab655

# or
git revert 2c4e6cb d95c7fb

… if the --sort=name error appears again, then it's a degraded version of tar that came with buildroot.

Anyway, thanks for entertaining this PR in the first place.

[n0rad]

Time to go to bed now, but tomorrow I will check what causes the issue and get back your commits.