nmfs-ost / ss3-source-code

The source code for Stock Synthesis (SS3).
https://nmfs-ost.github.io/ss3-website/
Creative Commons Zero v1.0 Universal
37 stars 17 forks source link

add gh action that does line by line ss_summary.sso comparison #458

Open k-doering-NOAA opened 2 years ago

k-doering-NOAA commented 2 years ago

Thanks @Rick-Methot-NOAA for this idea; this may help catch differences among models in quantities that are not tested

The difficult part is that there will always be some differences (e.g., the date of the model run may be different , the number of digits or use of sci notation or not may be different), so it may be difficult to make this job completely automated.

A first step toward figuring out how this could be useful, though, is simply to develop a job that does the comparison.

e-perl-NOAA commented 1 year ago

To do this I think it would have to run ss3 on a/the user example or test model(s) so that there would be something to compare. I guess the question is how complex a/the test model file(s) would need to be to compare all the important quantities. @Rick-Methot-NOAA do you have any more thoughts on this as we move issues over from the workflows repo to the other repos in the nmfs-stock-synthesis org?

Rick-Methot-NOAA commented 1 year ago

I am looking for a comparison that is functionally equivalent to what I do when I run a character-by-character, side-by-side text comparison and visually scan for color-coded differences that are meaningful. When I do that, my eye-mind is keying on significant digits. A difference in the 2nd or 3rd digit is a big deal, a difference in the 6th digit is not. Challenge is that the test needs to deal with a wide range of numbers and also sign differences (e.g. -0.000001 is not much different than +0.000001). So, I think we need to train the gha to do that numerically for every number in the output. Suggest starting by researching available algorithms to do such a comparison.

Rick-Methot-NOAA commented 1 year ago

I do not think we need to run this with every commit. Is is possible to create a gha dashboard so we could manually run when needed?

kellijohnson-NOAA commented 1 year ago

Yes, you can run it manually as long as we add workflow_dispatch: to on:.