Closed commandodev closed 3 years ago
I believe one problem is that when cachix went out of beta, the Free for Open Source cachixes were trimmed to 10 GB. The static-haskell-nix Cachix before had way more than 100 GB.
The my static-haskell-nix CI server has automatic cachix pushing enabled, but it builds against nixpkgs master every night, so it will create a large amount of builds. The /nix/store
on that is 1.5 TB currently, and all of that is pushed to cachix -- but only the most recent 10 GB are cached there, which explains bad cache hit rates.
For example, /nix/store/xrp57ywvgvnqwsg8plqgvz8c7pc4wbv3-ghc-8.6.5
exists on the CI server, so it was certainly uploaded to cachix in the past, but now is no longer there. I've manually re-uploaded it now.
In the future, it may make sense for me to offer the CI server directly as a binary cache, so that users can make use of the full 1.5 TB of builds, and not only 10 GB.
Finally, if you're using stack2nix
, then of course each resolver
may result in different packages. So it is expected that the GHCs are cached (which also take the longest time), but the individual packages not.
In summary, it's a cache size issue, not a reproducibility problem.
Thanks @nh2!
I've run cachix push
locally, so I had assumed that I would have pushed every dependency of rapid
to the cache. @domenkozar is that the right mental model here? I assume I must have done something wrong as the rapid cache only has 266.3 MB in it...
A little more investigating:
I've cleared the rapid cache and I've unchecked the upstream option. I've run
$(nix-build -A fullBuildScript --argstr projectDir $PWD)
And then
cachix push -j10 rapid /nix/store/mmqg7wiig0vyhy471hb8kmaplvdkvmab-rapid-0.1.0.0
(which is the resulting store path)
I'm now building on a separate machine (my nixos laptop), with cachix use {rapid,static-haskell-nix}
. Initial impressions are that it's building the world again.
Will report back when it's finished!
@commandodev Just in case: To check nix-level input reproducibility, you don't need to finish the build, just compare the rapid .drv
file paths that nix-build
prints it will build (and if different, nix-diff
them).
Depending of how you specify src
, it is easy for e.g. the .git
directory or similar nonreproducible files to make it in, and nix-diff
will help point out which those are.
I've not idea what was different between my machine and the run on github, but letting it run through and push from there seems to have worked:
https://github.com/cdodev/rapid/runs/1640209767 succeeded 1 minute ago in 4m 51s
I'd like to get this down a bit if possible, so going to look at caching /nix
(and perhaps the stack directory) itself to see if I can get any speed up
I've just enabled CI with cachix in my new project:
https://github.com/cdodev/rapid/runs/1636776075
Even though I've done
$(nix-build -A fullBuildScript --argstr projectDir $PWD)
followed by pushing to cachix locally, the CI run seems to be rebuilding the world.@nh2 would you expect the dependencies to be pulled from cachix in this case?
I ran this command locally: