The regression tests duplicate the responsibility of the Fortran model's test suite somewhat. The fv3gfs-fortran tests ensure that its results are bit-for-bit compatible through refactors when built in an identical environment. By also verifying the same checksums, the fv3gfs-wrapper tests essentially enforce build environment of the Fortran model and wrapper be the same. This is not easy to do with loosely coupled repositories.
Should we re-target these tests to instead ensure that the wrapper is bit-for-bit compatible with w/e version of fv3gfs-fortran is installed in the environment? The tests would look like
run fv3.exe
save checkusms
run wrapper
save checksums
compare 2 and 4.
This issue is blocking my proposed changes to use nix as package manager (see #268).
The regression tests duplicate the responsibility of the Fortran model's test suite somewhat. The fv3gfs-fortran tests ensure that its results are bit-for-bit compatible through refactors when built in an identical environment. By also verifying the same checksums, the fv3gfs-wrapper tests essentially enforce build environment of the Fortran model and wrapper be the same. This is not easy to do with loosely coupled repositories.
Should we re-target these tests to instead ensure that the wrapper is bit-for-bit compatible with w/e version of fv3gfs-fortran is installed in the environment? The tests would look like
This issue is blocking my proposed changes to use nix as package manager (see #268).