search

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.39k stars 1.3k forks source link

Docker container doesn't pin dependency versions #221

Open wwilson opened 6 years ago

wwilson commented 6 years ago

It looks like after this change the build container no longer pins the exact versions of every system package installed. It also appears to allow apt to pull in transitive dependencies, rather than specifying all packages explicitly.

This makes it a little bit harder to understand exactly what the build environment depends on, and also prevents our builds from being reproducible. In the worst case, you could imagine a situation where after all the testing has occurred during a release, Ubuntu upstream bumps a package version which causes a subtle change in behavior.

My guess is this was done because in practice, the team at Apple never rebuilds the container, but has a known good copy of it published somewhere? Is that right? Or was there some other rationale?

I think the two most straightforward solutions are either to pin the versions again, or to publish an "official" container, direct people to use it, and keep this file around mostly as documentation.

What do you think?

alexmiller-apple commented 6 years ago

I've felt kind of stuck between two difficult places here. I understand your sentiment that, particularly now that there will be frequent builds by various people of the docker image, it'd be nice to have everything pinned to a fixed version so that everyone gets exactly the same image.

On the other hand, I found it to be an incredible difficulty in upgrading the packages listed by a dockerfile that had all dependencies specified and versions pinned. The dependencies change across versions, new version restrictions start causing previously specified versions to cause conflicts, and now I'm left debugging why my packages form a cycle of blockers when I'm not even sure why most of these packages are even being installed.

I really doubt that I would be the only person having this problem though, but haven't had the time to go look around at how other projects specify their development dockerfiles in a maintainable way. I'd be interested in any links you have. Or, maybe the answer is just for us to publish a "blessed" docker image, and just set up a regular schedule of publishing updates to pick up new compilers and library versions over time?

wwilson commented 6 years ago

Yeah, I think an official container image is a totally reasonable solution to this problem.

FWIW, at one point we had a little script which given a list of desired packages (and optional versions) would create a temporary container, configure update channels, install the packages, then use dpkg queries to get a list of all installed packages and their versions (which necessarily constitutes a superset of the transitive closure of the dependency chain of the desired packages), and then regenerate the "apt install..." line that could be added back into the Dockerfile. So we just ran that when we had to make a change to the build environment.

I don't actually remember what that Python file was called, but grepping around in this repo I couldn't find it. Perhaps it's been lost to the mists of time. I doubt it would be that hard to rewrite.

alexmiller-apple commented 6 years ago

We discussed, and there's no team objection to pushing an official container image, so I'll go and do that.