RPi-Distro / pi-gen

Tool used to create the official Raspberry Pi OS images
BSD 3-Clause "New" or "Revised" License
2.64k stars 1.65k forks source link

Multiple builds which only differ in a later stage to cache erlier stages #796

Open MexHigh opened 2 months ago

MexHigh commented 2 months ago

I've added a stage "stage2a" which adds application specific things to the RPi OS Lite image (created by stage2). I want to create mutiple builds (optimally from a build pipeline) which only differ in the stage2a stage, meaning that everything up to stage2 remains unchanged.

Every stage starting from stage3 has SKIP and SKIP_IMAGE files in it so that the build process stops after stage2a. This works good.

However, when running multiple builds sequencially, stage1 and stage2 are never "cached" and rebuilt, which takes a lot of time. So my question is: Is there a way to cache build stages up to a specific stage to speed up multiple builds?. I'm using the docker build method (./build-docker.sh).


Edit: The instuctions in the README ("Skipping stages to speed up development") does not work for me. What I've tried (the correct stages already have SKIP_IMAGE files):

./build-docker.sh # builds all stages
touch stage1/SKIP
touch stage2/SKIP
PRESERVE_CONTAINER=1 CONTINUE=1 CLEAN=1 ./build-docker.sh

Both ./build-docker.sh calls run all build stages. Am I doing anything wrong?

pbaetens commented 1 month ago

for each build variant, just create a new stage folder?

MexHigh commented 1 month ago

But wouldn't that mean that a stage n+1 would include all changes made in stage n? I want to make e.g. five images, all with "unique" settings starting from stage 2 (Lite image).