Speedup system build time

[adeebshihadeh]

Goal is to get a <10m build time without a cache on the CI runner. According to https://github.com/commaai/agnos-builder/pull/251, we're currently at 33m.

For validation, trigger CI 5 times and post the timings of the "Build system" step. The max shouldn't exceed 10m.

1. remove anything that's unnecessary
2. refactoring and reordering steps
3. parallelize steps

If done carefully, it should be possible to do this without a comma 3/3X and have someone on the Discord test it for you at the end.

Bounty is worth 3x ($600) if it ends up being <5m.

[Emmanueltech]

Hi, I would like to work on this

[andiradulescu]

@Emmanueltech you need to show some initial progress in a PR to lock the bounty - the lock then times out after a week of no progress. More info in Bounties.

[robin-reckmann]

@adeebshihadeh I would propose to store the compiled .deb files for capnp, ffpmeg, modemmanager and qtwayland, as well as the wheel for pyqt5 in the repo. This should bring down the docker image creation time by quite a bit. Or do you prefer to compile them every time from source?

[adeebshihadeh]

@adeebshihadeh I would propose to store the compiled .deb files for capnp, ffpmeg, modemmanager and qtwayland, as well as the wheel for pyqt5 in the repo. This should bring down the docker image creation time by quite a bit. Or do you prefer to compile them every time from source?

I suspect it won't be worth it, but feel free to prove me wrong with a PR. Things I'd be concerned about:

• How do I figure out that I have to rebuild ffmpeg manually?
• There's a way to get the build time down enough without making this tradeoff.
• The build process for these checked-in binaries will break since they'll be run rarely.
• As a maintainer, I now have to rebuild these packages for PRs that want to update them (security risk).

[andiradulescu]

If we want custom "debs" built and uploaded separately, we should have a comma PPA.

The install of the comma PPA would look like this:

sudo add-apt-repository ppa:comma/comma
sudo apt update

Then you can just install the packages with apt.

Ubuntu launchpad also has it's own CI, so the packages will be built there.

Never actually used it, but this is nice way of delivering Ubuntu packages.

Could be used for:

• custom weston with comma patches and dependencies of weston
• custom debs we have in userspace/debs
• custom qtwayland
• custom ModemManager

As for these ones I would just move them to the 24.04 packages:

• ffmpeg - this will use mesa in mainstream kernel anyway - so there should not be an incompatibility with qcom OpenCL anymore
• capnproto - no idea why in 24.04 the version is 1.0.1 and latest is 1.0.2 (which we build) - we can definitely find out or live with 1.0.1

A random example of a popular PPA:

• https://launchpad.net/~kisak/+archive/ubuntu/kisak-mesa
• https://www.reddit.com/r/Ubuntu/comments/1cszy36/kisak_mesa_for_2404/

[andiradulescu]

If we don't want to deal with a PPA, we could:

• cache the "deb files" - when you want to update a deb file, you just change the deb file name (bump the appended version) and will just get rebuilt and stored in cache for the next run
• move all "deb files" to another repo, that will build them separately and push them to a GH Release - agnos-builder will just pull them from the Release of the other repo - this can have multiple separate releases for each package
• use a ccache that gets cached (similar to the kernel) for all building that happens in the docker container - this will reduce build times to a minimum, since almost no source file will get rebuilt, but pulled from ccache

[webbdays]

@adeebshihadeh is this issue still relevant?

[robin-reckmann]

@adeebshihadeh Can we ignore the package compile time assuming they will only be updated infrequently? With this PR https://github.com/commaai/agnos-builder/pull/354 the overall build_system.sh time is around 9mins (see https://github.com/commaai/agnos-builder/pull/359). This uses the docker cache for the package compilation, but builds the overall image from scratch for calculation of the build time.

[ilvar]

What is the config of namespace-profile-arm64-8x16-2004-caching runner?

[andiradulescu]

What is the config of namespace-profile-arm64-8x16-2004-caching runner?

exactly as in the name:

• arm64
• 8 CPU
• 16 GB RAM
• Ubuntu 20.04
• persistent docker cache