search

SolarArbiter / solarforecastarbiter-core

Core data gathering, validation, processing, and reporting package for the Solar Forecast Arbiter
https://solarforecastarbiter-core.readthedocs.io
MIT License
33 stars 21 forks source link

Integrate event forecasts into reports #380

Closed dplarson closed 4 years ago

dplarson commented 4 years ago

Creating a separate issue from #357 to make it easier to think through everything. (Or, at least, it's easier for me.)

Once the event obs/fx dataclasses are integrated, the next step is to make the reports support evaluating event obs/fx pairs. This should be relatively similar to the deterministic forecasts, but there may need to be a few (likely) minor tweaks. The main starting point seems to be reports.main.create_raw_report_from_data(), which:

  1. takes in report metadata (as a datamodel.Report() object) and data to be evaluated (a dict with Forecast and Obseration objects as the keys)
  2. pre-processes the data (validate + resample/align)
  3. computes metric values
  4. generates plots

Since #361 is already adding support for computing event metrics using the metrics.calculator.calculate_metrics() function, the key changes then should be in the pre-process step (which calls metrics.preprocessing.process_forecast_observations()). Namely, revising to:

  1. detect the forecast type (deterministic vs event vs probabilistic)
  2. add validation of the event fx (which is likely to be very minimal at this time)
  3. add resample and alignment of the obs/fx pair (which is also likely to be very minimal)

Once this issue is resolved, the focus will then turn to adding support for event forecast reports to the dashboard: https://github.com/SolarArbiter/solarforecastarbiter_dashboard/issues/216

dplarson commented 4 years ago

Based on prior discussions, the validation, resample and alignment steps will initially be just:

  1. confirm all time-series (obs and fx) are purely boolean valued
  2. align the obs and fx pair by timestamp (without any resampling)

with raises <Exception> added in relevant places for issues with data, e.g., obs provided at a different frequency than fx.

Then tests will be needed for the metrics.preprocessing.process_forecast_observations() (to check support for event obs/fx) as well as the reports.main.create_raw_report_from_data().

dplarson commented 4 years ago

Closing now that PR #361 was merged.