ajwheeler
commented
3 years ago We talked about this on a call and I want to more clearly articulate what I think is the way to go. I propose that we write some code to generate a plots comparing two versions of the code to each other and to external data from observations and other packages.
I'm imagining one script to generate a bunch of data which is run on two versions of the code, and another script that ingests those files and makes several comparison plots. Hopefully this can be automated to some degree. Because we would generate the data on the fly, long-term storage isn't as much of a concern, although it's still relevant for storing external data. I think JLD2 is appropriate for internal data.
I think at least these plots should be included (all with big residuals panels):
- a comparison to a solar spectrum with apples-to-apples binning and resolution
- for a few different sets of stellar parameters (these should also include spectra from MOOG or turbospectrum):
- a full solar spectrum
- zoom-ins to a few lines of interest: perhaps H-alpha, the the Ca triplet, etc.
- for the same sets of stellar parameters, but without comparison to other codes:
- plots of the number density of several species through the stellar atmospheres
- plots of the continuum and line opacities at a few atmospheric layers
I think that turning all this into a binary outcome ultimately wont be that helpful, since hopefully as the code improves our results will change for the better, and a picture is worth a thousand words.
andycasey
This issue is for discussing the design requirements for "gold standard testing" (since this is not necessarily obvious). I wanted to flesh out my own ideas and hear your ideas. As I write this out, I'm starting to think that we might not want to do this
High Level Description: The basic idea is to perform calculations under a standardized set of conditions and then cache the results (which serve as our "gold standard"). Then, after making changes we perform the same calculations and compare our result
Objectives: I would argue that there are 3 main objectives for this tool
High-Level Design: To be useful, I think our tools need to consist of 3 pieces
Other considerations:
Comparison to other testing approaches
"Gold standard" testing isn't the only tool at our disposal for verifying that our intermediate calculations give the correct result. Other types of tests include: 1. Comparing our synthetic spectra to the spectra for stars with known properties or the synthetic spectra produced by MOOG/Turbospectrum 2. There are some figures showing intermediate results of our calculations. For example, this is currently what we are doing with our continuum opacity calculations. 3. We could imagine extracting equivalent sections of code from MOOG/Turbospectrum and produce tables of expected results. Approaches 1 and 2 share the shortcoming that they don't provide strong constraints on all intermediate calculations. For example, consider a subdominant opacity source (that makes up say 1% of the total opacity). Approaches 1 and 2 could conceivably allow the result of the calculation to vary by 10%, in cases where the result should not have changed. Approach 1 also has the disadvantage that it's fairly expensive and it's not particularly useful for diagnosing where code got broken. Approaches 2 and 3 share the shortcoming that they are somewhat dependent on our choice of approximation. If our approximations differ from those used to produce the test answers, then the tests become less useful for addressing objective 1. In general, I would be much less keen on pursuing this "gold standard" approach, if approach 3 were easier to do. However, it's arguably the most difficult/error-prone because it would require us to significantly modify FORTRAN code. In general these other approaches provide a baseline for validating our "bootstrapped" answer tests.