search

ktindiana / sphinxval

SPHINX validation code for solar energetic particle models
MIT License
3 stars 3 forks source link

Feature Request: Capability to select only metrics applicable to a certain model via a (true) config file #49

Open ktindiana opened 9 months ago

ktindiana commented 9 months ago

Not all metrics apply to all models. With broader participation in the validation challenges and the start of cross-model comparisons, it is becoming worthwhile to make the code calculate ONLY the metrics relevant to a given model to avoid confusion or inappropriate comparisons.

A config file can be created that indicates which metrics should be calculated for each model (specified by short name or string within the short name). The config file must be accessible to the user in the base directory.

rickyegeland commented 5 months ago

Is this still relevant? Doesn't sphinx figure out what is possible to compute and what is not by inspecting the forecasts?

ktindiana commented 5 months ago

If a forecast field is present, SPHINX calculates all metrics for that forecast field. Depending on the model approach, only a subset of those metrics might actually make sense (e.g. only the comparison of the maximum value of the peak flux forecasted to the observed peak flux as opposed to comparing all peak flux forecasts to the observed peak flux).

The idea here is to output only the metrics identified as relevant to the model rather than dumping out everything and expecting users to ignore half of it.

rickyegeland commented 5 months ago

Discussed in meeting: a filter on the reports seems to be sufficient. @lukestegeman could take this one?