search

ocurrent / docker-base-images

Generate various Docker ocaml images
https://images.ci.ocaml.org
MIT License
29 stars 19 forks source link

Stagger image builds instead of building all of them at once #267

Open benmandrew opened 10 months ago

benmandrew commented 10 months ago

Currently we build the images on a weekly schedule, rebuilding all of them at once, which creates a large spike of demand on the cluster. It may be preferable to spread the builds out over the week.

We may wish to preserve the building-all-at-once behaviour for startup of the service.

shonfeder commented 7 months ago

It may be preferable to spread the builds out over the week

Could you elaborate a bit on the advantages of spreading them out? I can imagine some, but it would help us prioritize this issue to spell out the expected benefits :)

mtelvers commented 7 months ago

We often see failures when pushing to Docker Hub which appear to be related to multiple simultaneous pushes. Perhaps there is a rate or bandwidth limit? The failures need to be resubmitted manually. See https://github.com/ocurrent/docker-base-images/issues/211

shonfeder commented 7 months ago

We often see failures when pushing to Docker Hub which appear to be related to multiple simultaneous pushes.

Ah ha! That is a compelling motive for sure. But also automating the retries (perhaps after a delay?) also seems like it should be helpful per se, and maybe that should be tackled first?

shonfeder commented 3 months ago

Rather than introducing scheduling changes to stagger the builds, @mtelvers suggested today that we could just build each OS in sequence (and the version and architecture matrix under each OS will then fan out and take a good amount of our build capacity for most OSs).

This will reduce the pressure on the system and instead carry out the builds in a slower (likely steadier) stream.

I'm provisionally in favor of this approach, as it is a simpler way of achieving the desired aim here, and seems to have nice properties overall.