Automated builds - Githubissues

juanluisbaptiste commented 5 years ago

Hi @Conan-Kudo,

This is an issue to discuss the implementation of an automated cauldron build as being discussed in issue #12 .

As I mentioned on that issue, pushing cauldron images on a periodic basis to for ex, mageia:cauldron or mageia:latest, is not the best idea because you can't automate all the process: to add a new/updated image to docker's official library requires opening a PR against that project, which needs to be manually reviewed and accepted by the official library's team. We can go ahead and automate the build process and PR creation, but we would be adding an additional burden to them wich could mean that our updates take longer to be reviewed and accepted.

As an alternative we could use the mageia name but not as an official image from the mageia organization, but as the mageia docker hub user, which means we could push to mageia/cauldron, in a completely automated way, without having to update the official library.

WDYT ?

Conan-Kudo commented 5 years ago

Even a semi-automated process where we regularly update the official images should be set up. If people wanted fully automated builds, they can set up their own.

The point is that we need a process to ensure the official images are updated somewhat regularly.

Conan-Kudo commented 5 years ago

@tianon @yosifkit, would it be too terribly punishing for you guys if we did monthly submissions?

juanluisbaptiste commented 5 years ago

Even a semi-automated process where we regularly update the official images should be set up. If people wanted fully automated builds, they can set up their own.

The point is that we need a process to ensure the official images are updated somewhat regularly.

What's wrong with pushing to mageia/cauldron ? that one can be fully automated.

juanluisbaptiste commented 5 years ago

and mageia/cauldron would also be an official image (from the official mageia user).

Conan-Kudo commented 5 years ago

But it's not the same as mageia:cauldron, which is what people use now...

juanluisbaptiste commented 5 years ago

But it's not the same as mageia:cauldron, which is what people use now...

Many distros do it too, like CentOS for custom images like images for mariadb (centos/mariadb) or postgresql (centos/postgresql). The point is that pushing to mageia/cauldron can be don daily if needed, to mageia:cauldron will need to be more sporadic.

IMO, being cauldron the development version makes more sense to have builds updated as often as possible instead of once a month.

tianon commented 5 years ago

Monthly is perfectly reasonable for us! (Even weekly is generally fine.)

Conan-Kudo commented 5 years ago

If weekly is fine, I think we should do that.

Conan-Kudo commented 4 years ago

@juanluisbaptiste Not having regularly updated images for mageia:cauldron is hurting for leveraging some new features in Mock (it can use container images to bootstrap to a target environment for package builds). The currently published mageia:cauldron image is ridiculously old, lacking even DNF, much less latest RPM and whatnot.

Can we look to get this going soon?

juanluisbaptiste commented 4 years ago

@Conan-Kudo I'm sorry for being MIA lately, I'm at the last stage of a big project at work which meant working overtime including weekends for the last few weeks, and being honest I still will not have time until the end of next week that the project is delivered.

But I think we can get this moving forward now, as I have mentioned before in another issue, I had been working on automating this process with the build2release script. The script is ready but needs more testing and will take care of:

building the new image for the desired mageia version and architecture
Delete/recreate the dist branch so it only contains the last commit with the new images (and other images from other versions not being part of the build being done)
Optionally commit and push to this repo
Optionally create a PR to the docker-library repo (-U option, this one has not been very well tested and need hub program installed).

Read the header comments of the script to understand the image update process to get the image ready for docker-library as described to me by @tianon, which is the process build2release does.

So if you want you can try building the image using the script and then create a PR to this repo. I'll check it and if everything is ok I'll create the PR on docker-library. I don't have access to my mga vm to test it, but something like this should work:

./build2release.sh -a x86_64 -m 7 -M "Updated image for mga 7" -P

I think -d and -p options are not used/needed, last thing I was doing was a code refactoring around that. Let me know if you have any questions, and again I'm sorry for the lack of attention to this during the last few months.

juanluisbaptiste commented 4 years ago

Taking a quick look at the code, because of the current order of processing of cli parameter values maybe you will need to add -d even if -P is present too. Both options call prepare function, but if you are building planning on commit, you need the branch ready and that's what prepare function does.

juanluisbaptiste commented 4 years ago

Hi @Conan-Kudo ,

At last I had some time to work on this, I reworked completely the build script, it was over complicated to use and it did not work. Now I have a new version, I will push it shortly and start building new images too.

juanluisbaptiste commented 4 years ago

@Conan-Kudo Images pushed, I will do a test of them before updating docker/official-library and create the PR.

If you want to test them, you can downlad the rootfs files from dist branch, build them locally and run them.

juanluisbaptiste commented 4 years ago

@Conan-Kudo I tested both mga 6 and 7 x86_64 images and they work fine, I will be updating the official library later when I have some time.

juanluisbaptiste commented 4 years ago

Done, new PR docker-library/official-images#7784, @tianon could you check it out ?

juanluisbaptiste commented 4 years ago

Thanks @tianon, new images available.

juanluisbaptiste commented 4 years ago

@Conan-Kudo , so with the new build2relase.sh script to do a new build you would run this command:

./biuld2release.sh -b -p -U -v

Which will build the images (-b) while recreating the dist branch with only one commit with the new images, force push the new branch to this repo (-p) and update my fork of docker-library/official-images with the new commit id on dist branch (-U). The only thing missing would be to also create the pull request on docker-library/official-images, which can also be done using the github hub program.

With this tool I will be able to speed up image updates a lot and will be able to do it periodically as we discussed before.

Conan-Kudo commented 4 years ago

@juanluisbaptiste This is awesome!