Initial layer caching implementation

[GZGavinZhao]

WIP.

Current issues:

• [x] If you Ctrl+C or something catastrophic happens during layer creation (i.e. the fetching of dependencies), the incomplete layer is not deleted. It should be deleted. When this happens, find the hash at the start of the output of solbuild build and sudo rm -rf /var/cache/solbuild/layers/<hash>. Or just sudo rm -rf /var/cache/solbuild/layers to delete all layers.
• [x] It seems like 32bit dependencies are not installed correctly.
• [ ] Display the hash of the layer not only at the start of the build, but also at the end of the build as well.

This PR shouldn't touch anything outside of /var/cache/solbuild/layers, so if anything catastrophic happens, just run sudo solbuild dc and start over.

Test instructions:

1. Ensure you have the xxhash package installed because you need the xxxh128sum binary in your PATH.
2. gh pr checkout 95 or checkout this branch however you prefer to.
3. make
4. Now you should have a solbuild binary in ./bin/solbuild. Use this solbuild as how you would normally build packages. You should notice that after you build a package once, when you build it the second time, it should jump straight to the setup phase, skipping the step of installing dependencies.

[joebonrichie]

It looks like hashFile()/hashFileBytes() does the same thing as FileSha256sum()

I also tried an implementation with XXH3

import "github.com/zeebo/xxh3"

func FileXXH3Sum(path string) (string, error) {
    defer timeTaken("FileXXH3Sum")()
    mfile, err := os.Open(path)
    if err != nil {
        return "", err
    }
    defer mfile.Close()

    h := xxh3.New()

    _, err = io.Copy(h, mfile)
    if err != nil {
        return "", err
    }

    return fmt.Sprintf("%x", h.Sum128().Bytes()), nil
}
func timeTaken(name string) func() {
    start := time.Now()
    return func() {
        fmt.Printf("%s execution time: %v\n", name, time.Since(start))
    }
}

A similar thing could be done with blake3 which would allow 256bits again.

Dirty benchmarks (m64py) FileXXH3Sum execution time: 5.10317828s (uncached) FileXXH3Sum execution time: 1.441753051s (cached) FileSha256sum execution time: 9.536810397s (uncached) FileSha256sum execution time: 5.823365451s (cached)

edit: it looks like your hashFileBytes() implementation using io.Copy is slightly faster than the MapFile approach.

[ermo]

Friendly reminder that this solbuild version appears to rundep on xxhash, which contains the xxh128sum binary used to hash the layers.

Should probably be added in the cover letter or in the readme + added as a proper solbuild rundep once this hits the repo...

[GZGavinZhao]

Feature-wise complete, I've been using this since April as well. Need some code cleanup and add this feature behind a flag, then this should be ready for review.

[joebonrichie]

Chrooting after a interrupted build seems to fail, even after deleting the layers

^C 🗲  CTRL+C interrupted, cleaning up 
 ✗  Exiting due to interruption 
task: Failed to run task "build": exit status 1
[ble: exit 201]
⮞  ninya   git:main  ~/solus/…/l/libvpx  1  sudo rm -rf /var/cache/solbuild/layers/
 ⮞  ninya   git:main  ~/solus/…/l/libvpx  sudo solbuild chroot
 ✓  Using default profile name=unstable-x86_64
mount: /var/cache/solbuild/unstable-x86_64/libvpx/union: fsconfig system call failed: Stale file handle.
       dmesg(1) may have more information after failed mount system call.
 ✗  Failed to mount overlayfs point=/var/cache/solbuild/unstable-x86_64/libvpx/union err="exit status 32"
[ble: exit 1]

[GZGavinZhao]

@joebonrichie Arghh I completely overlooked the chroot command because I don't use it... I fixed the layer cleanup issue which caused the mount failure, but currently the layer is not being brought into the chroot so I'm going to fix that next.