Closed lboeman closed 2 years ago
@wholmgren This is getting close. I think a good way to help shake out bugs is to implement the dashboard/core side of things and try to get the whole thing working.
Before I move on to that, I wanted to get your thoughts on a few things:
I've added a exclude_system_outages
property to the report parameters that takes a boolean, and determines if we should exclude data from system outages during computation. Also added an "outages" key to the base report object that lists the report specific outages. Do you think that if the report_parameters have exclude_system_outages
set to True, we should automatically include the system outages in the outages
key?
Each outage object is very simple and has the form:
{
"report_id": <uuiid>, #report outages only
"outage_id": <uuid>,
"start": <datetime>,
"end": <datetime>,
"created_at": <datetime>,
"modified_at": <datetime>,
}
I think the algorithm for excluding the data will be something like:
Do you think that's a reasonable way to go about it?
Do you think that if the report_parameters have exclude_system_outages set to True, we should automatically include the system outages in the outages key?
Is the idea that if exclude_system_outages=True
, the report compute process would automatically pull the latest record of system outages? I thought we decided against implementing that for now, so I wonder if we really need that key at this point.
determine a set of unique forecast parameters for a report
In practice it might easier to just loop through the parameters for each forecast in a report instead of trying to be more efficient and only create the flagged series once for each parameter set. This would be more consistent with how preprocessing currently works.
for each flagged submission time store a (start,end) tuple to drop from analyses based on the forecast lead time and run length.
I thought we'd eventually need to get to a boolean series with this index:
Unclear to me if that's part of your plan in the last 2 bullets.
Use the report's cumulative set of (start,end) forecast period to drop observation data from the analysis, and report totals in preprocessing results.
We might need to drop both observation and forecast data. Dropping the observation would be sufficient to ensure the metrics are not computed. But the time series graph should not contain either for periods that have been removed.
To the extent possible, we should avoid overlap between the removed points and the rest of preprocessing results. For example, don't report nighttime values removed for a day that's not analyzed. This might be tricky so consider follow up work for this.
Is the idea that if
exclude_system_outages=True
, the report compute process would automatically pull the latest record of system outages? I thought we decided against implementing that for now, so I wonder if we really need that key at this point.
Yes, that was my idea. It seemed like it would be easier to add system and report outage tracking all in one shot, so I went ahead and added both of them in terms of storage and such. I can remove this parameter for now though, and we can use just report outages to start.
for each flagged submission time store a (start,end) tuple to drop from analyses based on the forecast lead time and run length.
I thought we'd eventually need to get to a boolean series with this index:
Unclear to me if that's part of your plan in the last 2 bullets.
Yes. That is the last step of the plan, with the overall process being: outage-ranges
-> forecast submissions times to drop
-> corresponding forecast values to drop
this intermediate list of start-ends is the step from forecast submission times to the corresponding time ranges to exclude. e.g. for a forecast submission time at 06:00 that falls within an outage period, a lead time of 1 hour and a run length of 3 hours we end up with (07:00, 10:00) start/end for that day. We'd also have to consider interval label here.
After getting all the forecast data ranges to drop we'd create a boolean series with the index you linked where between these (start,end) the values are dropped.
We might need to drop both observation and forecast data. Dropping the observation would be sufficient to ensure the metrics are not computed. But the time series graph should not contain either for periods that have been removed.
To the extent possible, we should avoid overlap between the removed points and the rest of preprocessing results. For example, don't report nighttime values removed for a day that's not analyzed. This might be tricky so consider follow up work for this.
It sounds like dropping both observation and forecast data prior to preprocessing will do this for us? I haven't looked carefully at where this would need to fit in in core, so I'm probably underestimating the complexity of implementing this.
Adds outage tracking for the system and reports.
Whole system outages are recorded using the admin cli.
Report outages are added by posting to a
/report/<report_id>/outages
endpoint.Outages are recorded as simple objects with a unique identifier and start and end parameters.