search

ktindiana / sphinxval

SPHINX validation code for solar energetic particle models
MIT License
3 stars 3 forks source link

Handling UMASEP False Alarms #12

Open ktindiana opened 1 year ago

ktindiana commented 1 year ago

After an SEP event starting, UMASEP will either forecast a series of Not Clear and may indicate Ongoing Event. After the observed event falls below threshold, that series of Not Clear forecasts continues and I bet this is what makes up most of the false alarms. There may be a better way to assess UMASEP performance that handles these "lagging" forecasts.

rickyegeland commented 12 months ago

This is challenging to handle in general. Ignoring these post-event Not Clears seems to imply defining "before event" / "after event" time windows (arbitrarily defined durations) and accounting statistics separately in these windows. That seems to lead to either a higher dimensionality contingency table and more complex metrics, or a series of parallel result sets.

rickyegeland commented 12 months ago

Steve's idea of producing a validation keyed only on triggering events (e.g. flare > given magnitude) would give a new result set that is not impacted by this model behavior.