switch to new Debian stable + build standard images on GitHub

[lsf37]

This switches the docker builds to the new Debian stable release (bullseye) and adds github tests for the main images.

The images are built sequentially instead of concurrently, because they build on each other.

(Part of seL4/ci-actions#84)

[LukeMondy]

Just from a quick look at the logs, I'm guessing this error is because Debian has released bullseye, and so pulling things from testing has changed a lot. ;)

[lsf37]

Oh hi @LukeMondy, cool to see you commenting! I was just thinking about switching everything over to bullseye since that is now stable.

Any gotchas I should be aware of before I embark on that project?

riscv-gcc is part of stable now, which is very nice, currently looking if qemu-riscv is also there. Would be cool if we didn't need an extra variant any more for those.

[lsf37]

riscv-gcc is part of stable now, which is very nice, currently looking if qemu-riscv is also there. Would be cool if we didn't need an extra variant any more for those.

# qemu-system-riscv64 --version
QEMU emulator version 5.2.0 (Debian 1:5.2+dfsg-11)

😄

[LukeMondy]

Oh hi @LukeMondy, cool to see you commenting! I was just thinking about switching everything over to bullseye since that is now stable.

Definitely recommend to switch to stable ASAP, it just makes a lot of things easier.

Any gotchas I should be aware of before I embark on that project?

Best to try and get as many packages in stable as possible. Pulling from testing and unstable adds complexity and potential conflicts, that can come and go quite a lot. A big part of the Debian snapshot stuff, and surrounding build infra, is to kinda smooth over the bumps they can produce - but best to avoid them at the source if possible.

Unfortunately, due to the relentless march of time, as the current stable release gets older, more and more packages will have to come from testing/unstable, since newer versions won't be available in stable - hence my recommendation to move to stable ASAP - you get the most time with the least pain ;)

[lsf37]

Cool, that meshes with my thinking. So far I've been able to remove a lot of stuff that was dealing with older (buster) things. Let's see how it goes.

[lsf37]

Not entirely smooth sailing, but some progress. Main points so far:

• we should upgrade compilers to gcc-10 and clang-11. Possibly more warnings, but that should be generally a good thing
• riscv tool chain is now there, but only for gcc-8. That is a bit annoying (different version to the rest, and it was on gcc-10 for our own build), but probably fine.
• python2 is becoming a problem (some packages only available for python3 now). I think it might be time to officially drop python2 support. It's been after EOL for a while. Not exactly sure where all our dependencies on python2 are, though.

[lsf37]

Ok, I think this is now ready for review. I'm still going through all the CI GitHub actions to see if anything unexpected breaks (a few PRs for compiler update so far), but it seems to be going reasonably well so far.

[lsf37]

This is still in testing, because we're seeing a failure of the CAmkES simulation tests with RISC-V. Likely to be fixed on that end, though.

[LukeMondy]

Just some more thinking for the V2 - the Bamboo setup had two builds:

1. Someone pushes a change to the code: Using the last known good snapshot date, build the changes that someone has asked for.
2. Once a day: try building using today's date as the snapshot date - if it works, then update the last know snapshot date.

The idea was to try to avoid confounding things - there are two distinct things that can break building the images: someone's changes to the code; and using different snapshot date (because maybe something changed upstream).

The benefit of splitting them means that: a) you know what went wrong, and b) even if the daily build breaks, you are not blocking people making changes using the last know good snapshot date.

It worked... fairly well, I think? However, the issue can occur where the daily build breaks, and stays broken (as in, something upstream has changed enough that the scripts need modifying, but no-one has fixed it yet). People can still make code changes, because the last known good snapshot date still works fine - but at some point, they're going to need a newer version of a package, and so the snapshot date will need to be bumped forward. At this point, if the daily build has not been fixed, they are now blocked. You can go months with a broken daily build, but it just means more pain when you need to fix the daily build :sweat_smile:

It probably won't be an issue soon, because you'll be pulling almost everything from stable, but as time goes on, testing and unstable are needed more and more, and changing snapshot dates can have a big effect.

[lsf37]

I think that was a good plan and we can replicate it fine on GitHub.

The normal pull request test will take the last known good snapshot. On push to master, we also take the last known good snapshot and deploy with that. And separately, we do a scheduled run, once a day that updates the snapshot and commits the new snapshot date. That commit should trigger a normal test and then deploy. (Or we deploy directly and add something to not trigger the normal test on that snapshot date update commit).