Consider keeping EOL distros for build reproducibility

[richardbarlow]

Would it be possible to keep the EOL distro builds, rather than removing them from Docker Hub when the distro goes EOL? Is the reasoning/policy of removing the EOL distro builds documented anywhere?

I am using the images built from this repo in a CI environment to build firmware images for a product. Having the ability to reproduce old builds for maintenance and support purposes is critical. It is unfortunate that the reproducibility depends slightly on the host distro and, of course, BitBake tries to keep this dependency to an absolute minimum. However, this does mean that it is not always possible to build a given Poky release from an arbitrarily new host distro. It is therefore very important that older host distro images are available to be able to keep building images for older products.

I appreciate that an EOL distro means that it is no longer going to receive the normal level of support and maintenance upstream, however that doesn't mean that it is made unavailable. One can still download a very old version of a distro and run it if desired/necessary. For my particular use case I see the images built from this repo akin to the ability to download an old version of a distro out of necessity. Only keeping images for the very latest/supported distro releases makes them too much of a moving target and unfortunately not useful for my purpose.

Would you consider keeping EOL distro image builds, rather than removing them when adding newer releases? It would be a shame if I had to duplicate/extend the effort put into creating these images all for the sake of keeping a few images hanging around.

P.S. Right now I have a build that requires the fedora-30 image, which was removed 6 days ago. Due to a perfect storm of changes (GCC 10, glib 2.31, python2 removal) using fedora-31/32 requires a significant amount of work to rebase to a newer Poky release and perform product verification.

[rewitt1]

Hi @richardbarlow,

My first question would be whether it would be possible to use one of the LTS distros such as Ubuntu 18.04?

Regarding preserving EOL distros, although I don't like the idea of using EOL software, we could perhaps preserve them in a separate repository on dockerhub such as "crops/poky-eol". The EOL images would not be tested as a part of CI nor would they ever be updated.

All that being said, I still highly recommend building any images used for production processes locally.

[leehambley]

I second @richardbarlow 's request. I'm currently learning Yocto and trying to work from a vintage 2016 tarball provided by the vendor that I have lovingly re-factored into a yocto tree using submodules. Everything is ancient, and poorly documented, and I was happily using the :debian-8 tag of the crops/poky image (the oldest in the repo, last week when I was working, and also the only version that actually worked).

I'm trying to upgrade things, but it's 4 years of software updates I have to trawl through, and I limited git sources from the vendor (shallow clones, packed in tarballs) so I'm on very, very thin ice.

I am cloning the debian-8 image to my own systems now for safety's sake, but I was surprised to see this project's apparently policy of trimming old image tags. (Is there some effect such as max tagged versions or similar which insentiences trimming the tag list?) Regrettably, I didn't even see a Git tag, let alone a Docker tag for which version in this repo was used to build that particular container image, so I think I just have to freeze a tarball of the Docker image in place, and that becomes my "gold master" whilst I work on updating all the parts of the release.

[rewitt1]

@leehambley,

(Is there some effect such as max tagged versions or similar which insentiences trimming the tag list?)

There is nothing so sophisticated. Trimming the EOL images has always happened manually, this time I did it when I added new Fedora images. And the trimming was already long overdue considering debian-8 was actually removed as part of 82367e20ae55fbba7877475d101e0b084bdef38e back in September.

The images on dockerhub were never intended to be releases in the traditional sense. If anything they were convenience images for experimentation and general usage.

Regrettably, I didn't even see a Git tag, let alone a Docker tag for which version in this repo was used to build that particular container image

There are no tags because there are no releases other than tracking the master branch. The images are built when new changes are pushed to github and also once a week to pick up updates. The images are pulling packages from the respective distro repositories. Without keeping track of the versions of each package in an image, and then subsequently adding that to the build of the images, the package versions in the image will always be whatever the latest packages are when the image is built. Which means that the version on github used to build the image will only impact some portion of the image.

However, another tag of the form crops/poky:committish that corresponds to the committish of the most recent build may be useful. But there is no archiving that is occurring.

so I think I just have to freeze a tarball of the Docker image in place, and that becomes my "gold master" whilst I work on updating all the parts of the release.

For any image on dockerhub other than the base/signed images, I recommend building it locally if needed for production. It ensures that you know exactly what is in the image, and that it was built in an environment meeting whatever level of security you desire.

With that disclaimer, here is what I'll do moving forward:

1. Create a new repository on dockerhub crops/poky-eol where the EOL distros will be located
2. Add the most recently removed images in 481b63321d4d42326d3d0b753d852d761df21e67(and debian-8) to the crops/poky-eol dockerhub repo
3. Any future EOL images will be added to crops/poky-eol using the most recently built image

I'm also considering creating an eol branch on github l that tracks master, but where the EOL dockerfiles aren't removed. The idea would be that it might be a bit easier to build the EOL distros locally after they have been removed. But since that can also be achieved by using git history, I'm not sure of the value. I'm also trying to limit the maintenance work as much as possible.

I'm also sorry this disrupted some workflows. But this kind of feedback always helps steer things.

[rewitt1]

There is now a crops/poky-eol on dockerhub: https://hub.docker.com/repository/docker/crops/poky-eol

[richardbarlow]

Thanks for creating crops/poky-eol @rewitt1! That's perfect. I hadn't realised that the images were being rebuilt on a regular basis to pull in distro updates. I can appreciate that means that my desire for reproducibility is being slightly compromised so it's good to know. However, I still think it's useful to have these images as a starting point and keep product development moving forwards at a reasonable pace. Once we have more time available we will probably transition over to a builder image fully under our control.

Also, thank you for your work maintaining these images; they have been invaluable in quickly getting products operational and shipped :)

[leehambley]

Hi @rewitt1, outstanding responses thanks. I appreciate everything you have done to help support our less than optimal misuse of the crops images :)