Increasingly masked rolling horizon

[brynpickering]

Problem description

Our current rolling horizon method (operate run mode) doesn't really allow planning for e.g. seasonality. We might want to think of the horizon as high resolution over the next 48 hours (hourly), then daily resolution (e.g. mean/max of the hours) for the few weeks after that, then weekly/monthly/annually after that, depending on total time horizon.

It would allow future seasonality to be captured, even though at low resolution (indicative of our understanding of the climate, if perhaps not the weather). As the window rolls forward, information becomes clearer. This is particularly handy when considering long term storage.

This wouldn't really affect the window aspect of the operate run mode.

Steps to introduce functionality

• [ ] Work out how to add the horizon points at which resolution changes into the YAML format
• [ ] Work out how to efficiently update a model with a newly masked time horizon on each run iteration. At the moment, we can just update some Pyomo params with the new horizon data, without having to rebuild the model, because the number of timesteps stays the same. This would involve a different number of timesteps on each iteration...

[brynpickering]

An implementation was started in https://github.com/calliope-project/calliope/tree/windowsteps_matrix

[brynpickering]

Some thoughts on this:

We could have a list of horizon periods for each window, e.g. in YAML:

config.init:
  operate_window: 24h
  operate_horizon:
    - offset_start: 0
      offset_end: 47
    - offset_start: 48
      offset_end: 215 # 1 week later
      resolution: 24h
      agg_method: mean
    - offset_start: 216
      offset_end: 945 # 1 month later
      resolution: 1w 
      agg_method: mean  
    - offset_start: 946
      resolution: 1m
      agg_method: mean  

Then, every 24h window there would be a masked horizon of 2 days at full res, one week at 24hr res, one month at weekly res, and the rest at weekly res. For a year @ hourly resolution, this would lead to the maximum number of timesteps in a window being 48+7+4+11 = 70. This would reduce as we go along as e.g., there will be fewer months in the future as we progress through the months.

Maybe agg_method can't be in there as we have use different methods to resample our variables at the moment. Some need summing and others averaging.

[brynpickering]

If we loaded this directly from file using #532 then it would require a table of the form:

	2005-01-01 00:00	...	2005-01-02 01:00	...
2005-01-01	10	...	10	...
2005-01-02		...	10	...
...	...	...	...	...

Where rows are windowsteps and columns are timesteps. It would be a triangular matrix, with row-wise timestep data being non-NaN from and beyond each windowstep.

To get different resolutions, the rows would be more sparsely populated (e.g., for a monthly resolution horizon period there would only be data on the 2005-XX-01 00:00 timesteps).

Then on running the model, we iterate through each windowstep and re-construct the optimisation problem with the new timeseries data.

[brynpickering]

Loading from file with pre-resampled data allows the underlying data to change per windowstep. This would allow us to represent poor quality seasonal and subseasonal forecasts where the aggregated data beyond e.g. 5 days is off and the model only finds this out once the window steps forward and that same period is then e.g. 1 day away.

Perhaps less useful (or, at least, I can't think of a use case) is being able to have variable window lengths.