Closed jschwab closed 6 years ago
My changes may not be consistent with your ruby style or mesa_test design. Refactor as needed.
Is the default behavior for most users to create their own checksums? When I talked with Frank, it sounded like most users were just running each_test_run_and_diff. If that's the case, I'd rather require as few options for the bulk of users and have those few that make checksums have to put in a --no-diff option.
I'll defer to your experience. The code style looks fine. Either way, this looks better than relying on an environment variable being set.
each_test_run_and_diff is run by more people (~10 at each release). each_test_run is run more often (~nightly by Bill).
That probably argues for --no-diff. I'll update my PR.
There are two main MESA testing scripts: each_test_run and each_test_run_and_diff.
Checksum files (checks.md5) are used to confirm that results are bit-for-bit identical.
each_test_run does not assume these checksums are up-to-date. Thus it generates them when doing ./rn and then checks against them when doing ./re.
each_test_run_and_diff does assume that these checks are up-to-date and checks against them when doing ./rn and ./re.
Before a release, someone does ./each_test_run and confirms things are working. Then they commit the updated checksums and folks go forth and do ./each_test_run_and_diff on many different machines to confirm that everyone is getting the same results.
However, during development, the checksums are not kept up-to-date.
This changes mesa_test to emulate each_test_run by default. If the --diff argument is given, it emulates each_test_run_and_diff.
The goal is to prevent a lot of (noisy) failures that are only because the checksums aren't up-to-date and not because MESA has a problem.