Get paths faster - Githubissues

nix-community / cache-nix-action

Cache Nix store in GitHub Actions to speed up workflows [maintainer=@deemp]

MIT License

29 stars 6 forks source link

Get paths faster #3

Closed deemp closed 11 months ago

deemp commented 11 months ago

Currently, I use nix copy to copy paths from a cached chroot store from and into root store.

Copying takes a lot of time at the start of a job.

Possible solution

daemon 1 with low priority copies paths that aren't yet in /nix/store.
daemon 2 with high priority substitutes paths on demand.
Daemons run during all steps of a job until after saving cache to GitHub Actions cache.
If a daemon 2 needs to substitute and lock the store, daemon 1 stops for a while, then continues.

Advantages

Get necessary paths selectively.
Utilize time of other steps (there are 2+ CPUs - docs).

Disadvantages

Other steps may require all CPUs

Alternatives

Cache and restore full store.
- macOS runners don't support restoring a root store from cache. (I couldn't, TODO docs link)
- macOS runners don't support chroot stores for building and running (docs)
Use a local binary cache store via file:// (docs)
- Triggers a lot of copying from substituters (don't know why)
https://github.com/DeterminateSystems/magic-nix-cache-action/issues/16

deemp commented 11 months ago

Solved by https://github.com/deemp/cache-nix-too/tree/use-nix-quick-install-action as it becomes possible to restore and save store paths directly without copying from an intermediate store.