search

COSIMA / access-om3

ACCESS-OM3 global ocean-sea ice-wave coupled model
13 stars 6 forks source link

CI reproducibility test #41

Open aekiss opened 1 year ago

aekiss commented 1 year ago

It would be good to have a CI reproducibility test, as this is fundamental to regression testing but isn't guaranteed https://github.com/COSIMA/access-om3/issues/40

The test would be to cold-start two identical runs and see if the restarts are identical. The runs would need to be multi-processor but could otherwise be cheap (e.g. 1 day at 1° resolution), with restart saved at the end of each run.

aekiss commented 1 year ago

@micaeljtoliveira MOM6 has a REPRO build option https://youtu.be/94m3CMTwJ1E?t=415

Is this equivalent to what we do in our build system?

aekiss commented 8 months ago

@CodeGat and @jo-basevi have built a very nice model repro CI framework https://github.com/ACCESS-NRI/access-om2-configs/pull/2

This will be put in a separate GitHub actions repro so it can be reused for repro testing of other model configs, including ours.

CodeGat commented 8 months ago

Hello! Yes, we have been working on testing out the rest of the repro-ci tests for ACCESS-OM2 and putting more generic actions into https://github.com/ACCESS-NRI/reproducibility. There will more generic actions to port from access-om2-configs (I can already see some good candidate steps and actions) but it would be great to see how it goes with ACCESS-OM3 as well! We will be finishing up and putting it through its paces around next week :)

dougiesquire commented 7 months ago

I think it could be a good idea to prioritise this. We've now had a few instances where config-breaking PRs have been merged.

From a very quick peruse, I think the main thing stopping this from just working for us is that we need some ACCESS-OM3-specific code for extracting checksums from the output of a model run (e.g. https://github.com/ACCESS-NRI/access-om2-configs/tree/main/test). This should be pretty straightforward