search

getsolus / solbuild

The Solus package build system
https://getsol.us
Apache License 2.0
21 stars 12 forks source link

Root layer caching #68

Open GZGavinZhao opened 10 months ago

GZGavinZhao commented 10 months ago

This is the design for caching our container root between subsequent builds on the same package. The container root will be reused as long as system.base, system.devel, and the package's dependencies remain the same. This will eliminate most if not all the time needed for fetching and installing dependencies.

To do this, we will need to have three overlays that are mounted and unmounted one after another. Relative paths are relative to /var/cache/solbuild/<profile>/<package>/.

  1. Overlay 1: for ensuring system.base, system.devel, and other packages needed by solbuild to function are at their latest version.
    • lower: img
    • upper: base
    • merged: unionb
    • other mounts:
      • eopkg package cache: /var/cache/eopkg/packages
    • <hash-of-dependencies> is generated inside this overlay after everything is up-to-date.
  2. Overlay 2: for installing the dependencies of the package to build. This overlay is only mounted if the layer doesn't exist already.
    • lower: img:base
    • upper: /var/cache/solbuild/<layers>/<hash-of-dependencies>. This is the layer that will be cached and reused.
    • merged: uniond
    • other mounts:
      • eopkg package cache: /var/cache/eopkg/packages
  3. Overlay 3: for actually building the package
    • lower: img:base:/var/cache/solbuild/<layers>/<hash-of-dependencies>
    • upper: tmp
    • merged: union
    • other mounts:
      • everything that we currently mount

<hash-of-dependencies> is generated by ypkg-install-deps --dry-run --json (getsolus/ypkg#57).

ReillyBrogan commented 9 months ago

What about creating different overlay layers for each individual package install? That would allow layers to be reused between builds.

GZGavinZhao commented 9 months ago

I'm concerned about file conflicts between different packages. If we install packages one by one, we would need extra measure to make sure file conflicts are reported to us, while if we let eopkg install everything eopkg will handle that.

I used to mention running triggers is also an issue, but I guess we could force run triggers before running ypkg-build, so now my main concern is if packages have file conflicts.