MESAHub / mesa_test

Command line tool for running the MESA test suite and uploading results to MESATestHub
MIT License
2 stars 3 forks source link

Add --diff option to test_one and test_all #5

Closed jschwab closed 6 years ago

jschwab commented 6 years ago

There are two main MESA testing scripts: each_test_run and each_test_run_and_diff.

Checksum files (checks.md5) are used to confirm that results are bit-for-bit identical.

each_test_run does not assume these checksums are up-to-date. Thus it generates them when doing ./rn and then checks against them when doing ./re.

each_test_run_and_diff does assume that these checks are up-to-date and checks against them when doing ./rn and ./re.

Before a release, someone does ./each_test_run and confirms things are working. Then they commit the updated checksums and folks go forth and do ./each_test_run_and_diff on many different machines to confirm that everyone is getting the same results.

However, during development, the checksums are not kept up-to-date.

This changes mesa_test to emulate each_test_run by default. If the --diff argument is given, it emulates each_test_run_and_diff.

The goal is to prevent a lot of (noisy) failures that are only because the checksums aren't up-to-date and not because MESA has a problem.

jschwab commented 6 years ago

My changes may not be consistent with your ruby style or mesa_test design. Refactor as needed.

wmwolf commented 6 years ago

Is the default behavior for most users to create their own checksums? When I talked with Frank, it sounded like most users were just running each_test_run_and_diff. If that's the case, I'd rather require as few options for the bulk of users and have those few that make checksums have to put in a --no-diff option.

I'll defer to your experience. The code style looks fine. Either way, this looks better than relying on an environment variable being set.

jschwab commented 6 years ago

each_test_run_and_diff is run by more people (~10 at each release). each_test_run is run more often (~nightly by Bill).

That probably argues for --no-diff. I'll update my PR.