Incremental package builds

[aparcar]

Hi all,

looking at the phase2 buildworkers it looks like they rebuild all packages on a daily basis, using whatever SDK is available by that time. If I'm not mistaken, this introduces three pitfalls:

• All packages are daily rebuild which requires a lot of time, even if only a few packages, may none, changed at all
• A package in version 1.0 and release 4 may change from one day to the other based on different buildsystems. So even if it is the same release, it's maybe different package due to (possibly breaking) changes in the underlying build system.
• Packages are builds with different buildsystem revisions, which makes it harder to reproduce them. For example when testing for reproducibility.

An alternative could be to rebuild only what's changed since the last build. The following two shell command give a list of all packages changed since the last build of packages.git. This can be used as an input for whatever build setup is used:

ARCH=x86_64
LAST_BUILD="$(curl \
  https://downloads.openwrt.org/snapshots/packages/$ARCH/feeds.conf \
  | grep " packages " | cut -d '^' -f 2)"

PACKAGES="$(git diff --diff-filter=d --name-only $LAST_BUILD \
  | grep 'Makefile$' | grep -Ev '/files/|/src/' \
  | awk -F/ '{ print $(NF-1) }' | tr '\n' ' ')"

The package index creation currently happens on the worker, which would no longer be of much use as the worker no longer stores all packages locally. As the signing of packages via signall.sh happens on the control server anyway, it may as well run make package/index after the new packages where synced over.

This should dramatically reduce compile times for packages and would even allow a faster iteration than 24h, is desired.

[aparcar]

Related build: store buildsystem revision in packages

[aparcar]

We could collect package versions of both "tools/" and "toolchain/" and whenever something updates, a treewide package release rebuilds all packages.

With the current system package version stays the same even ig GCC is upgraded from 8 to 9 (or 10), which seems something to happen soonish.

[neheb]

@aparcar is this still valid?

[aparcar]

Well sure, incremental builds are still a thing.