Deoverlapping for All Clear

[ktindiana]

Add deoverlapping techniques for All Clear validation.

For triggered forecasts:

• Select all forecasts generated from the same flare/CME trigger
• Choose an "any" result for hits, false alarms or "all" result for misses, correct negatives

As a first attempt for continuous forecasts:

• Make three categories of time periods - quiet in prediction window, SEP event in prediction window, ongoing SEP in prediction window
• Bin quiet time periods into 24 hours, daily time period
• For each quiet 24 hour time period, choose an "any" result for false alarms and an "all" result for correct negatives
• For SEP time periods, choose an "any" result for hits and an "all" result for misses
• Exclude forecasts during ongoing SEP time periods

[rickyegeland]

I like the approach for continuous forecasts, but would opt to make the quite time period bin width equal to the prediction window of the model – assuming the models are sane enough to keep that fixed. In any case, it would be good to be able to customize that bin width.

[rickyegeland]

Quiet time periods will generally not divide into integer N bin widths, so there will be 1 or 2 non-standard size bins for each quiet segment, depending on your choice for selecting the start time of the bin edges. I think this is OK given that N >> 2 most of the time.

[rickyegeland]

This approach should be extended to Probability forecasts as well, choosing to take the max probability in the quiet bins, as well as for SEP matches. The per-bin statistics (min, max, mean, std) of the model probability are also interesting quantities that indicate the model's robustness and self-consistency. A huge spread of probabilities in a bin indicates that the model produces a huge range of outputs that make it challenging for operators to trust any given single forecast.