Future validation: Models as ensemble – all predict large/small events, likelihood of large/small event?

[ktindiana]

For the SEP Scoreboard, how can we look at the models as an ensemble to better inform how operators use them. Do the models together give us more reliable information than a model alone? For example:

• If all or most of the models are predicting that a large event will occur, does that mean we can be pretty sure there will be a large event?
• What if the models are predicting that a small or below threshold event will occur, can we trust that?
• Are there cases where models have had widely spread forecasts? What have been the observed outcomes for those?

Are there features, reporting, or a workflow we can add to SPHINX to evaluate this?

[rickyegeland]

The looks like a kind of post-analysis study, driven by the regular per-model sphinx outputs. I am imagining a script that reads the output files and generates new, aggregate forecast jsons that can be run through sphinx in a second pass.