Open CodeGat opened 5 months ago
Also pinging @harshula for his thoughts
I think it is worth seeing if it is even still possible to rebuild exactly: don't need to check spack.lock
is reproduced, just check if it even works to exactly rebuild from spack.lock
.
Or maybe build from spack.yaml
and seeing if we can recreate spack.lock
? If not, why not? If hashing changes with every spack
version then this is probably not worth the effort.
If we can't re-build exactly from spack.lock
do we need to generate a PR against a known test config to test for bit repro? Would we then update the spack.lock
if we reproduce?
There was a desire to update the software stack and test for bit repro (External Dependency CI). Is this effectively that process?
Background
In a similar vein to the scheduled Bitwise Reproducibility testing on
ACCESS-NRI/access-om2-configs
, we want to test that, on a schedule, rebuilding different versions of models from scratch produce the same build. Since we usespack
, a good hallmark of a reproducible model build would be thespack.lock
file.Potential Solutions
on.schedule.cron
, get the list of tags that we care about (similar toaccess-om2-configs
with theconfig/released-configs.json
), then get the associated GitHub Releases Assets, download thespack.lock
and compare this against thespack.lock
of a rebuild of the model at this version on Gadi.Questions
spack.lock
file, it will always produce the same build. See the bottom of https://spack.readthedocs.io/en/latest/environments.html#creating-a-managed-environment - we are guaranteed to produce the same build. If this is so, this seems like a vacuous check, then. We would be comparingspack.lock
against itself if we do aspack env create my-env spack.lock && spack env activate my-env && spack install
- thespack.lock
will not change.spack.yaml
file, it may well not produce the exact samespack.lock
, ever. The concretization logic is arcane wizardry in which the things we specify in thespack.yaml
will not change, but the dependenices may well change with the whims of the concretizer. It may still do the exact same thing on the tin, just with a patch version bump to some random dependency. In this case, this workflow will almost certainly always fail. In which case...Pinging @aidanheerdegen