Time structure must move along when rolling #99

Closed spine-o-bot closed 3 years ago

In GitLab by @Tasqu on Sep 16, 2019, 03:43

Original comment from #model in Slack: "It seems that at the moment, the temporal_block and rolling time structures are entirely separate, so that the underlying time structure created by temporal_blocks is simply viewed through the rolling windows in parts. However, at least for some applications using time-step aggregation, it would be required that the structure constructed using the temporal_blocks moves along with the rolling window."

Further comment from #model in Slack: "The main reason we implemented time step aggregation in our Backbone model was to extend the horizon of each optimization so that we could handle unit commitment and hydropower scheduling at the same time. Unit commitment requires hourly (or better) time resolution, but extending an hourly model to cover for a full year or even multiple years required for hydropower (or any long-term storage) scheduling is infeasible. Thus, we started to aggregate the time steps for later in the horizon, since hydropower scheduling works fine even with week or month long time steps. In order to get hourly results for the entire year, the aggregated time structure must move along with the rolling horizon, since the results at the aggregated time steps are not useful as results."

This feature is required for case study A4 #46

In GitLab by @manuelma on Sep 17, 2019, 13:06

Related to Spine-project/Spine-Database-API#44

A little bit of history: initially, we wanted the temporal blocks to be defined for just one window, and pushed forward by whatever was defined as the rolling horizon structure. I think it was me who decided to make the temporal blocks fixed and just view them through the rolling horizon structure. This decision was mainly motivated by the fact that temporal blocks are defined in absolute terms, so it seemed weird to push them in the model.

Having said that, I don't see any problem on taking the other approach, except for that annoyance: we will have temporal blocks defined absolutely in the data that then will become something else in the model. Is there a middle way here?

In GitLab by @mihlema on Sep 17, 2019, 15:21

How about the following idea:

temporal_block contain start_datetime and end_datetime (in absolute, thus meaningful)
the rolling Object specifies mainly the general characteristics of the rollinghorizon window (reoptimization_frequency, initial_condition_duration, rolling_window_duration)
on the relationship temporal_block__rolling, we define the parameter timestep_duration which is then used to define the temporal aggregation of the rolling_window with respect to the specific temporal_block

Apart from that - food for thoughts:

should the rolling horizon itself have a horizon_start_date and horizon_end_date (current implementation) or will this result from the temporal_blocks?
we should have a good fall back to the original temporal structure, if a user doesn't need a rolling horizon (no need for a rolling object in that situation)

In GitLab by @manuelma on Sep 17, 2019, 15:40

Another one:

Drop the rolling object and replace it with a mandatory and unique model object (or similar name). This model object would have mandatory parameters start_date_time and end_date_time that need to be there regardless of whether or not a rolling horizon is used. The same model object would have optional parameters reoptimization_frequency, initial_conditions_duration, and rolling_window_duration, that can be used to define a rolling horizon optimization.
Modify the temporal_block object so start_date_time and end_date_time parameters are relative to the start of the current window (or the only window if no rolling horizon is defined).

I think this would be clear for most users. Just define the temporal frame of your model in the model object and then refer to that frame when defining your temporal_blocks.

In GitLab by @mihlema on Sep 17, 2019, 16:03

Just for clarity:

Modify the temporal_block object so start_date_time and end_date_time parameters are relative to the start of the current window (or the only window if no rolling horizon is defined).`

How exactly do you picture this relative to the current window/I don't seem to the the current window yet.
This setting could potentially be cumbersome in cases where we don't roll, but have temporal_block starting at different points in time

In GitLab by @manuelma on Sep 17, 2019, 16:10

How exactly do you picture this relative to the current window/I don't seem to the the current window yet

Well in my mind we would roll according to what's defined in the model object (parameters start_date_time, end_date_time, reoptimization_frequency, initial_conditions_duration, and rolling_window_duration). This would produce 'windows' with an absolute start and end. We would combine that start and end with the relative starts and ends in our temporal blocks to compute the absolute starts and ends of our temporal blocks in the 'current' window. And we run spine model from there.

This setting could potentially be cumbersome in cases where we don't roll, but have temporal_block starting at different points in time

You think so? All one needs to do is provide the start and end times of the blocks relative to the start of the only window (no rolling, so the entire horizon becomes the only 'window'). Not too different from specifying start and end times in absolute terms imo.

In GitLab by @pvper on Sep 18, 2019, 06:06

Can't you have both? If you set the temporal_block start_date_time to a duration, it will be relative to the current window, if you set it absolute it will be absolute. Same with the end_date_time.

Ex: Hydro power optimization, during the winter the reservoir balance is not that important since you have a lot of play and small inflow, so in the period between 2018-11-01 to 2019-04-01 you want the balance resolution to be daily or weekly. but when the spring flood comes you want it to be hourly. In this case you want the temporal blocks to have an absolute start and stop time regardless on where you are in the rolling.

But the electricity balance you want to have resolution that follows the rolling, these would then use a temporal block with duration instead of datetime for the start and end parameters.

In GitLab by @manuelma on Sep 18, 2019, 11:00

Ok so what about this:

Keep the rolling object as it is today> If present, SpineModel knows it has to do a rolling horizon optimzation.
Keep the temporal_block objects as they are now, with absolute start and end that cannot be overriden by the model when doing a rolling horizon optimization.
Introduce a new rolling_temporal_block object with parameters relative_start and relative_end (or similar names), taking a duration value. These are moved with the rolling window.

So, if the user doesn't want to do a rolling horizon optimization they just need to define (absolute) temporal_blocks and don't worry about anything else.

If the user wants to do a rolling horizon optimization they have to define the rolling object, and they can have both absolute temporal_blocks that don't move with the window, and rolling_temporal_blocks that do move.

In GitLab by @manuelma on Sep 18, 2019, 12:08

assigned to @manuelma

In GitLab by @mihlema on Sep 19, 2019, 12:12

@pvper interesting example. However, I don't think it would work the way you proposed it: The rolling horizon window would break the optimization problem in e.g. daily optimizations. Thus the resolution of the water balance equation would be at lowest 1 day (unless the electricity balance would be completly decoupled from the water flow, which is probably not the case).

An alternativ, for the example of @pvper without the need for an extra object rolling_temporal_block:

1 rolling object for the complete year (e.g. 1 day reopt_frequency)
1 temporal_block for the electricity system
2 temporal_blocks for the water flow (1 winter/1 summer)
the timeslice_duration (which may vary), could be defined e.g. on the temporal_block__rolling relationship

In GitLab by @pvper on Sep 19, 2019, 12:39

I see, I'm not very familiar with the current implementation so I probably mix up the concepts. I guess I'm talking about a moving window, where to optimization is ex. one year that is always pushed forward.

In GitLab by @manuelma on Sep 19, 2019, 12:44

An alternativ, for the example of @pvper without the need for an extra object rolling_temporal_block:

1 rolling object for the complete year (e.g. 1 day reopt_frequency)

1 temporal_block for the electricity system

2 temporal_blocks for the water flow (1 winter/1 summer)

the timeslice_duration (which may vary), could be defined e.g. on the temporal_block__rolling relationship

Interesting, so how do you make the electricity block roll and the water block stay fixed?

In GitLab by @mihlema on Sep 19, 2019, 12:54

The water block would still move along with the rolling, which is inevitable if we break the optimization after every day for the electricity system. For winter, the timeslice_duration for winter temporal_block would be set to 1 day whilst for summer temporal_block2 it would be hourly to be exact.

One could consider that the timeslice_duration could be given a even lower resolution than the rolling_window (e.g. 1week instead of 1day, and SpineModel takes care of "cutting/alligning" the resolution.

In GitLab by @jkiviluo on Sep 19, 2019, 14:01

Per's example is really good, because it forces us to think how we do this in a way we won't regret later.

To me, it seems like we need two kinds of temporal_blocks. Those that define relative properties and move with the rolling window. And those that define absolute properties and stay put. In the hydro case, you don't need to think about two different systems (water and electricity) - it can just be electricity where the reservoirs and water flows are expressed in MWh. Still, you want to have a high resolution in the first hours of the rolling window and high resolution during the spring flood. When the simulation moves forward, at some point these high resolution periods will overlap. And the highest resolution should be used. E.g. the absolute spring flood period is defined as 3-hourly and the first 24-hours of the moving window are hourly. When these start to overlap, then the 3-hourly gets to be hourly.

Of course we don't want to make things complicated for the simple case (e.g. single full year optimization with uniform resolution). I would think that this could work, if one connects these different temporal_blocks to the model object and the API figures out what to do. There are possible outlier cases that need to be considered. E.g. you have a 2-week rolling window and an absolute period for the spring flood. Between them, there would be non-defined time. This should be captured with an error - I suppose.

What to call these? Not sure, but we will need to have a model_instance object sooner than later, so I would consider using that in some way. But maybe it's just by connecting temporal_blocks to it. And should there be separate absolute_temporal_block and relative_temporal_block for clarity or is it a switch inside temporal_block or even interpreted from which parameters have been used (absolute_start vs relative_start). I might prefer the separate class names.

Also, when it comes to defining the different resolution stages in the rolling window, one should be able to say just the duration of each stage instead of relative_start and relative_end. The problem with the latter, is that you have to change values in all stages, if you want to change the duration of the first stage. However, that requires the establishment of order between the rolling window temporal_blocks (and we don't support that yet).

In GitLab by @jkiviluo on Sep 19, 2019, 14:23

Just went back to the original issue and I have to say I still like the clarity of the original proposal (https://gitlab.vtt.fi/spine/data/issues/44) where there is a separate time_frame defining the absolute frame that will be modelled (start and end). Then one can connect rolling window objects to that (and taking cue from the discussion here, also absolute temporal sections/blocks). And the word block would be better used for the block of nodes that will get the same temporal definitions. Word section is better for temporal stuff, IMHO (I've tried to read their meanings from dictionaries and think which one is better where).

In GitLab by @mihlema on Sep 20, 2019, 07:50

Proposal @manuelma and I discussed yesterday:

Object classes:

temporal_block
- - Parameter:
- - start_datetime (date_time)
- - end_datetime (date_time)
- - time_slice_duration (duration; single value or Array)
rolling
- Parameter:
- - reoptimization_frequency (duration; single value)
- - initial_condition_duration (duration; single value)
- - rolling_window_duration (duration; single value)

Relationship classes:

temporal_block__rolling:
- Parameter:
- - time_slice_duration (duration; single value or Array)

Remark:

the rolling object doesn't need a start/end_datetime as this will result from the connected temporal_blocks

The usage would be the following:

If a user doesn't use a rolling horizon he just defines all parameters in the temporal_block (Note: in particular the time_slice_duration parameter!). No rolling object is required.

If the user uses a rolling horizon, the relationship temporal_block__rolling is used to define the time_slice_duration (Note: this then corresponds to the resolution within the rolling window).

If for a temporal_block the parameter time_slice_duration is defined for both the object class and the relationship class, the time_slice_duration of the relationship class has priority.

Attached is a file how the given example could look like in the proposal:

temporal_example.sqlite

In GitLab by @Poncelet on Sep 20, 2019, 08:58

Overview of discussed use cases:

Simple optimization (no rolling)
Non connected periods (no rolling) (e.g., different representative days for production costing/investment planning)
Rolling horizon where the resolution is constant with respect to the rolling window (or put differently, the resolution is not fixed for a given absolute time interval), e.g., Topi's example where the first hours of each optimization window are modeled at high resolution and the later hours at a lower resolution
Rolling horizon where the resolution depends on the absolute time information
Hybrid forms of 3 and 4

In GitLab by @manuelma on Sep 20, 2019, 10:29

So @mihlema's proposal is basically the same as the current implementation: temporal blocks are defined in absolute terms and never move. The rolling object specifies rules for splitting these blocks into several windows of a rolling horizon optimization.

The only extra ingredient is the relationship rolling__temporal_block and its parameter time_slice_duration. This is used to define a resolution which is specific to the rolling window and overrides that of the temporal block. Thus, it can be used to specifically solve the problem presented in this issue. (Note: maybe I want to call that parameter time_slice_duration_override to be perfectly clear).

Use cases 1 and 2 above can be implement using only temporal_block objects, the way we already know.

Use case 4 can be implemented using temporal_blocks and a rolling object.

Only use cases 3 and 5 require the rolling__temporal_block relationship.

Personally speaking I'm almost ready to go with this proposal, but I'd also like to hear comments and perhaps an even better idea...

In GitLab by @mihlema on Sep 20, 2019, 10:45

As a side remark: The specific representation/implementation of 2 is discussed in #124

In GitLab by @DillonJ on Sep 23, 2019, 09:25

Nice discussion - seems like all the reasonably common structures are supported - quite neat.

The only thing that I think would be nice to support is to have a temporal structure that represents a generic year, without having to specify the year... is this possible?

In GitLab by @mihlema on Sep 23, 2019, 12:53

@jody Thanks for the input! In my opinion, the generic year representation is somewhat related to the representative days representation. I changed the description of #124 a bit, so we can discuss on this topic there.

In GitLab by @mihlema on Sep 23, 2019, 13:30

As a follow-up on Fridays call, I am trying to give an example of implementation for every use case (1-5) for my proposal. I renamed time_slice_duration with rolling_resolution for the temporal_block__rolling relationship (open to any renaming suggestions).

Proposal

`rolling` object

The (possible multiple) instances of the rolling object class contain only the properties of the rolling horizon window(s). Note the time-resolution information is given in the temporal_block (static) and/or the relationship temporal_block__rolling (moving along).

The rolling object has the following mandatory parameters:

rolling_window_duration: the duration of each rolling window (duration value)
reoptimization_frequency: how much do we roll forward each time (duration value)

The rolling object also has the following optional parameter:

initial_condition_duration: how much do we look back to get initial conditions from the result of the previous window (duration value)
Temporal_block object

The (possible multiple) temporal_block objects define the temporal structure of the model. Each temporal block provides information to compute a series of ordered time-slices (which are used as time indexes in the model).

The temporal_block has the following mandatory parameters:

start_date_time: the start of the block in absolute (date_time value)
end_date_time: the end of the temporal block, either absolute (date_time value)

The temporal_block has the following optional parameters:

time_slice_duration: the duration of the time slices in the block (duration value). This establishes points in the temporal block where one time slice ends and the next starts (equivalent to the block's time-resolution). This parameter needs to be defined if the resolution is independent of the rolling window. Note that duration values can be arrays, which means temporal blocks can have varying resolution.
Relationship temporal_block__rolling

The relationship between a temporal_block and a rolling object is mandatory in case of a rolling horizon. Through this relationship the start and end of the rolling_horizon is readout (if multiple are connected, the first start and the last end).

The relationship has the following optional parameter:

rolling_resolution: the resolution information equivalent to the time_slice_duration but for the rolling window. (duration value). This parameter needs to be defined if the resolution is dependent of the rolling window. Note that duration values can be arrays, which means temporal blocks can have varying resolution.

How it works, generally speaking

If no rolling object is created, but only a temporal_block: The optimization will run as a single shot optimization.

If a temporal_block is connected to a rolling object but the parameter rolling_resolution is not specified: The optimization will run according to the rolling window but retrieving the resolution information from the temporal_block’s time_slice_duration.

If a temporal_block is connected to a rolling object and the parameter rolling_resolution is specified: The optimization will run according to the rolling window and retrieve the resolution of the parameter rolling_resolution

1. Single Optimization

Example: One year e.g. 2020, hourly resolution, no rolling

The user needs to define the following object/ parameters:

1 Temporal_block (e.g. year2020)
- start_datetime (2020-01-01 00:00:00)
- end_datetime (2021-01-01 00:00:00)
- time_slice_duration (1h)

2. non-connected periods

Example: 2 years, 5 representative days per year

10 Temporal_block (e.g. year2020_1, year2020_2,…, year2021_5)
Each temporal_block has e.g. for year2020_2:
- start_datetime (2020-05-21 00:00:00)
- end_datetime (2020-05-22 00:00:00)
- time_slice_duration (1h)
- weight: 73
Remark: Is this user friendly? -> #124 on the implementation of representative periods

3. Rolling horizon with variable resolution within the rolling window

Example: one year (2020), 1 day rolling window with a re-optimization frequency of 12h; rolling window with variable resolution: first 4 hours in half-hourly resolution, then in hourly resolution

1 Temporal_block (e.g. year2020)
- start_datetime (2020-01-01 00:00:00)
- end_datetime (2021-01-01 00:00:00)
1 rolling (e.g. daily_rolling)
- Reoptimization_frequency (12h)
- Initial_condition_duration (e.g. 0h)
- Rolling_window_duration (24h)
1 temporal_block__rolling relationship (year2020, daily_rolling)
- rolling_resolution ([30 min, 30 min, 30 min,… 1h, 1h] Comment: adds up to 24 hours)

4. Rolling horizon where the resolution depends on the absolute values:

Example case: One year optimization, first quarter of the year in daily resolution, rest of the year is in hourly resolution; rolling window duration 2 days, re-optimization frequency 24h

1 Temporal_block (e.g. year2020)
- start_datetime (2020-01-01 00:00:00)
- end_datetime (2021-01-01 00:00:00)
- time_slice_duration([1d, 1d, 1d,1d … ,1h, 1h])
1 rolling (e.g. rolling2)
- Reoptimization_frequency (24h)
- Initial_condition_duration (e.g. 0h)
- Rolling_window_duration (48h)
1 temporal_block__rolling relationship (year2020, rolling2)

5. Hybrid form

Example: Absolute information for temporal_block_static, while temporal_block_dynamic changes resolution with the rolling_window

1 Temporal_block (e.g. temporal_block_static)
- start_datetime (2020-01-01 00:00:00)
- end_datetime (2021-01-01 00:00:00)
- time_slice_duration([1d, 1d, 1d, 1d … ,1h, 1h])
1 Temporal_block (e.g. temporal_block_dynamic)
- start_datetime (2020-01-01 00:00:00)
- end_datetime (2021-01-01 00:00:00)
1 rolling (e.g. rolling2)
- Reoptimization_frequency (24h)
- Initial_condition_duration (e.g. 0h)
- Rolling_window_duration (48h)
1 temporal_block__rolling relationship (temporal_block_static, rolling2) Not obligatory
1 temporal_block__rolling relationship (temporal_block_dynamic, rolling2)
- rolling_resolution ([30 min, 30 min, 30 min,… 1h, 1h])

The rolling optimization starts/ends with the first/last temporal_block(s) connected to it via a relationship.

There are rules that need to be specified on how to treat cases where "static" temporal blocks & rolling windows are not in sync (crucial cases: 4 & 5; needed for any of the rolling proposals). Comment: For the simplicity of the example, they are chosen in sync.

In GitLab by @manuelma on Sep 23, 2019, 14:16

Same as @mihlema above, here's another proposal and a description of how it satisfies requirements.

Proposal

Rolling object

The singleton rolling object provides all the necessary information to compute the start and end of a series of rolling windows. Note the rolling windows don't have any time-resolution information (that is given by the temporal blocks, below).

The rolling object has the following mandatory parameters:

(horizon_)start_date_time: the absolute start of the rolling horizon (date_time value)
(horizon_)end_date_time: the absolute end of the horizon (date_time value)
rolling_window_duration: the duration of each rolling window (duration value)
reoptimization_frequency: how much do we roll forward each time (duration value)

The rolling object also has the following optional parameter:

initial_condition_duration: how much do we look back to get initial conditions from the result of the previous window (duration value)

Temporal block object

The (possible multiple) temporal_block objects define the temporal structure of the model. Each temporal block provides information to compute a series of ordered time-slices (which are used as time indexes in the model).

The temporal_block has the following mandatory parameters:

start_date_time: the start of the block, either absolute (date_time value) or relative to the start of the rolling window (duration value).
end_date_time: the end of the temporal block, either absolute (date_time value) or relative to the start of the rolling window (duration value).
time_slice_duration: the duration of the time slices in the block (duration value). This establishes points in the temporal block where one time slice ends and the next starts (equivalent to the block's time-resolution). Note that duration values can be arrays, which means temporal blocks can have varying resolution.

How it works, generally speaking

If the rolling object is defined, it means Spine Model must run a rolling horizon optimization. If not, then Spine Model runs a 'single-shot' optimization.

In case a rolling object is not defined, then all time-slices from all temporal blocks having absolute start_date_time and end_date_time are included in the 'single-shot' optimization. Temporal blocks having relative start_date_time and end_date_time are unusable without a rolling object, so an error or warning should be raised if this occurs.

In case a rolling object is defined, temporal blocks having absolute start_date_time and end_date_time are split and their time-slices allocated to the different rolling windows, following a set of rules yet to be defined.[^1] Meanwhile, temporal blocks having relative start_date_time and end_date_time are moved along with the rolling window, which means their absolute start_date_time and end_date_time are computed for every rolling window by adding the relative values to the window start (using period arithmetic). One can imagine that the blocks are being pushed forward x units of time at each roll, where x is exactly the reoptimization_frequency.

How it specifically satisfies the requirements

Simple optimization (no rolling)

Use absolute temporal_blocks (with start_date_time and end_date_time specified as date_time values) and no rolling object.
Non connected periods (no rolling) (e.g., different representative days for production costing/investment planning)

Same as 1.
Rolling horizon where the resolution is constant with respect to the rolling window (or put differently, the resolution is not fixed for a given absolute time interval), e.g., Topi's example where the first hours of each optimization window are modeled at high resolution and the later hours at a lower resolution

Use relative temporal_blocks (with start_date_time and end_date_time specified as duration values), together with an appropriate rolling object.
Rolling horizon where the resolution depends on the absolute time information

Use absolute temporal_blocks (with start_date_time and end_date_time specified as date_time values), together with an appropriate rolling object.
Hybrid forms of 3 and 4

Use both absolute and relative temporal_blocks together with an appropriate rolling object.

[^1]: How do we allocate time-slices from absolute temporal blocks to rolling windows? Here are two possible options: (A) If a time-slice fits entirely within one window, then allocate it to that window; if it spans over two or more windows, then split it and allocate each resulting sub-time-slice to the corresponding window. (B) Allocate each time-slice to all windows it overlaps, even if that means the same time-slice is included in two or more windows.

In GitLab by @jkiviluo on Sep 23, 2019, 18:50

I have trouble with some of the design choices in the above alternatives. I think there should be a clear distinction between the actual time frame that is to be modelled and the definition of the windows with changing resolutions. Keeping these separate clarifies a lot (however, I'm open to doing a short-hand for simple modelling cases). Also the rules that follow from the above proposals are quite complex. Another issue is that I don't see how the geographical information fits in: we might want to model different systems in different resolution. Hence:

Yet another proposal

Time frame

The task of the time_frame is to establish the absolute time that is to be modelled. Including both the source dates (or years) as well as the target dates (or years). It is cumbersome to change the dates, if the absolute time is also in the rolling window (I use 'optimization_window' here for the 'rolling', sorry) or in the changing resolution time sections. Also, the time frame is shared by the whole model_instance whereas the more elaborate stages of optimization window can be properties of specific groups of nodes.

It should also be possible to just list the source years and target years (instead of the cumbersome full datetime). Leap year is a bit of a challenge and probably needs some rule (or actually a method or methods that can be chosen).

time_frame could possibly handle single shot optimizations fully. However, if we allow this, I think defining e.g. duration both in time_frame and in optimization_window should raise an error. I don't like behaviors that one has to know about.

Time frame has the following parameters:

reoptimization frequency: how much do we roll forward each time (duration value)
start_date_time: the absolute start of the rolling horizon (date_time value)
end_date_time: the end of the temporal block, either absolute (date_time value) or relative to the start of the rolling window (duration value).
optionally years_to_be_modelled: could be used to replace the date_times, since it's much nicer to just write '[2020, 2025, 2030]' when that's what you want to do
we should also consider the target vs. source years, but can be discussed later

Optimization window

An optimization_window defines the structure of the rolling window or, in case of a single shot, the structure of the non-rolling time. And these can be applied separately to different 'groups of nodes'.

optimization_window has the following parameters:

duration: the duration of the block (duration value).
resolution: the resolution the block (duration value).
These two could also be combined into resolution_for_duration if it's easier to use (a dict array)
could potentially hold reoptimization frequency if that would make sense

How it works, generally speaking

time_frame defines the time span that will be modelled. If time_frame does not define reoptimization_frequency then it's a single shot and time_frame could define resolution (but could also be defined in an optimization_window - just if there is a dual definition, then an error should be raised).

If there is an reoptimization_frequency then an optimization_window is required to define the roll. I'd like to see node_groups to be used to define potentially separate optimization_windows when needed, but this can be left out for now. Similarly the model_instance can be left out for now, if we just assume that the time_frame applies to the whole database.

The optimization_window holds only relative information. Relative from the start_date_time as defined in time_frame. This way the whole temporal structure can be changed to a new time position with ease.

There can also be optimization_windows that define a 'static' position in time to allow accommodating things like a spring flood. However, these are also relative, but relative from the first day of the year or similar measure (e.g. first Monday of November). I guess we could also allow to assign an absolute start_date_time, but that won't move with the start_date_time of the whole model and is a potential problem.

When the roll moves forward by reoptimization_frequency, the whole optimization_window structure moves too.

How these satisfy the requirements

Simple optimization (no rolling)

time_frame defines the absolute time (and can be arrayed to run multiple alternative years) while one time_section connected to a node_group covering the whole model scope would define the resolution. Alternatively we could allow time_frame to have also resolution as long as we get an error when resolution is defined also in time_section.
- time_frame("full_2018").start_date_time = 2018-01-01 00:00:00
- time_frame("full_2018").start_end_time = 2018-31-12 23:59:59
- time_frame("full_2018").resolution = 1h
Non connected periods (no rolling) (e.g., different representative days for production costing/investment planning)

Same as 1. time_frame could define an array of representative time periods or optimization_window could be used to define the 'stay put' time segments. I think I'd prefer the former.

time_frame("full_2018").start_date_time = [2018-01-01 00:00:00, 2018-05-01 00:00:00, 2018-11-01 00:00:00]
time_frame("full_2018").start_end_time = [2018-01-07 23:59:59, 2018-05-07 23:59:59, 2018-11-07 23:59:59]
time_frame("full_2018").resolution = 1h

OR

time_frame("full_2018").start_date_time = 2018-01-01 00:00:00
time_frame("full_2018").start_end_time = 2018-31-12 23:59:59
optimization_window.from_year_start = [0 5m 11m] // how many hours/days/weeks/months from year start
optimization_window.duration = [1w 1w 1w]
optimization_window.resolution = 1h // could also be [1h 1h 1h]

Rolling horizon where the resolution is constant with respect to the rolling window (or put differently, the resolution is not fixed for a given absolute time interval), e.g., Topi's example where the first hours of each optimization window are modeled at high resolution and the later hours at a lower resolution

time_frame defines the absolute time and the optimization_window defines the moving window with stages. E.g. the left "heat" branch in the figure above.

time_frame("full_2018").start_date_time = 2018-01-01 00:00:00
time_frame("full_2018").start_end_time = 2018-31-12 23:59:59
time_frame("full_2018").reoptimization_frequency = 3h
optimization_window("for_electricity").duration = [24h 24h 1w]
optimization_window("for_electricity").resolution = [1h 3h 24h]

Rolling horizon where the resolution depends on the absolute time information

time_frame defines the absolute time and the optimization_window, with counting from e.g. 'year_start', would define the resolution windows. Why 'year_start'? It enables to start the model from middle of summer and still have the spring floods in the right place. We can have other similar conventions like 'model_start'. And one could also use absolute time references. I just find them cumbersome and bit dangerous, if you want to change e.g. the year.

time_frame("full_2018").start_date_time = 2018-01-01 00:00:00
time_frame("full_2018").start_end_time = 2018-31-12 23:59:59
optimization_window("spring_flood").from_year_start = 2880h // how many hours/days/weeks/months from year start
optimization_window("spring_flood").duration = 2w
optimization_window("spring_flood").resolution = 3h
optimization_window("for_electricity").duration = [24h 24h 1w]
optimization_window("for_electricity").resolution = [1h 3h 24h]

Hybrid forms of 3 and 4

Number 4 is already a hybrid.
Model with different time resolution structures in different parts of the model

time_frame.start_date_time = 2018-01-01 00:00:00
time_frame.start_end_time = 2018-31-12 23:59:59
time_frame.reoptimization_frequency = 3h
optimization_window("for_electricity").duration = [24h 24h 1w]
optimization_window("for_electricity").resolution = [1h 3h 24h]
optimization_window("for_heat").duration = [24h 72h]
optimization_window("for_heat").resolution = [3h 6h]

In GitLab by @DillonJ on Sep 24, 2019, 10:28

I think we need to decide what we want to achieve with this. As I understand it (and please correct me if I am wrong @mihlema and/or @manuelma ) @manuelma's current proposal is an incremental change to the current structure and the proposal is completely developed to the point that it can be implemented.

If we want to include the geographical dimension to this, I think we have a lot more discussing to do and it will take a much longer time to get to the point of implementation.

We need to get something implemented relatively soon so we can close out the case studies. The actual problem we are trying to solve here is clear and well defined, that the time structure should move when rolling and the current proposal addresses that. Perhaps we need to consider a two-step implementation? Or at least consider simple modifications to @manuelma's proposal that leaves the door open to inclusion of the geographical dimension?

In GitLab by @manuelma on Sep 24, 2019, 11:22

I think there should be a clear distinction between the actual time frame that is to be modelled and the definition of the windows with changing resolutions.

That could be done if it brings more clarity. For example we can move in my proposal horizon_start... and horizon_end... from the rolling object to some model object.

Also the rules that follow from the above proposals are quite complex

Which ones in particular? Maybe we should discuss more in detail. We are trying to cover a large number of very different use cases with a minimal setup and hence the complexity.

Another issue is that I don't see how the geographical information fits in

What about relationships node__temporal_block and unit__node__direction__temporal_block? Is that what you mean? These are not included in the discussion to focus on the current issue, but are already part of Spine Model.

In GitLab by @mihlema on Sep 24, 2019, 11:39

I do fully agree with @DillonJ and @manuelma on this. I think both @manuelma and my approach are very similar to what we have now. If there is not a very good reason for a change, I would not completely re-do the whole time structure.

Main functionality differences the other two proposals:

@manuelma proposal seems to be more intuitive for the most of us
@mihlema proposal opens the possibility for more flexibility as we could have e.g. different rolling_window properties sequentially and in parallel, as we can support more than one rolling object (Example: year one 1d re-optimization frequency, year two with 2d re-optimization frequency; less common cases)

I'm slightly in favor of the extra flexibility, but I can understand if simplicity outweighs the extra functionality. I will also update my description above to make it hopefully more understandable.

In GitLab by @manuelma on Sep 24, 2019, 12:26

I think "my" proposal (which actually owes a lot to @pvper's comments) could also accept multiple rolling objects the way @mihlema envisages. Also I'm not opposed to give @jkiviluo's proposal full consideration but it's also true that it has the one big disadvantage that it doesn't build up on what we already have.

In GitLab by @Poncelet on Sep 24, 2019, 19:38

I agree with Jody, Maren and Manuel. There must be good reasons to abandon what we have in this stage (which does not fully excludes later refactoring.

That being said, below an attempt to get to the essence of the different proposals. To allow a more easy comparison, I renamed Juha's terms (i.e., time_window = rolling, time_section = temporal_block)

Proposal Maren:

temporal scope of the simulation is represented via the temporal_block objects
resolution of the simulation is represented via the temporal_block object. A single exception is Topi's case where the resolution is relative to the window (in that case, resolution is defined via the rolling__temporal_block relationship)
rolling object only holds information on how to roll (how much to move forward, initial conditions, etc.)
rolling is explicitly initiated by the rolling__temporal_block relationship

Proposal Manuel:

temporal scope of the simulation is either represented via the rolling object (or alternatively a model object) or via the temporal_block object
rolling object contains information regarding the temporal scope and regarding how to roll
resolution of the simulation is always represented in temporal_block objects (and it can be chosen to have resolution relative to rolling window, or absolute)
rolling is explicitly initiated by the presence of a rolling object

Proposal Juha:

temporal scope of the simulation is represented via the rolling object (time_section in Juha's terminology)
resolution information is covered in temporal_block objects
temporal_block objects are by default relative to rolling object
every simulation is by default a rolling optimization, i.e., rolling is implicitly initiated if the duration of the time steps in subsequent temporal_block objects do not cover the temporal scope as provided by the rolling object.

In GitLab by @Poncelet on Sep 24, 2019, 20:09

Regarding the benefits and drawbacks of different proposals:

Proposal Maren versus proposal Manuel:

Maren's proposal is unambiguous regarding where the temporal_scope of the simulation is defined (i.e., in the temporal_block objects)[^1]. In Manuel's proposal, the temporal scope can be defined either in the rolling or in the temporal_block objects. I think this can lead to confusion (as I think Juha pointed out): sometimes the temporal_block object defines the entire scope of the simulation, sometimes just a fraction of it. Also, this leads to more complex rules to handle the different situations (e.g., what happens when absolute temporal block objects are defined and a rolling object is defined so both having absolute start_datetime and end_datetime parameters).
Manuel's proposal is unambiguous as to where the resolution is defined (always in the temporal_block objects - but the resolution can be defined relative to the temporal_window and in absolute terms). In contrast, in Maren's proposal, the resolution is normally defined in the temporal_block object unless it is relative to the rolling horizon window.

In my opinion, having a clear and unambiguous scope of the simulation is more important than the same for the resolution. In that regard, I would currently favor Maren's proposal.

Proposal Juha

I see some potential problems with this proposal:

Need 2 objects for the simple single-shot optimization
As temporal_block objects (time_sections) are by default relative to the rolling object (time_frame), it is not clear to me how different representative periods would be modeled. As suggested, one option would be via the start_datetime/end_datetime parameters of the rolling object holding arrays. However, what if we would want to have different representative periods that use different resolution (or have a different resolution pattern within)? How to indicate which temporal_block object(s) corresponds to which representative period?
it is unclear to me how the different temporal_block objects are related to each other (given that they do not rely on absolute temporal information). For instance, in the figure, there are 4 temporal_block objects: two of them have a relative parameter value of 1, and two have a relative parameter value of 2. How do we know which 2 follows which 1?
unclear to me where the information regarding how to roll is stored?
every optimization is by default a rolling_horizon. This means the user can unintentionally start a rolling_horizon...

[^1]: Although it can be defined via a combination of different temporal_blocks (either sequential or overlapping)

In GitLab by @Poncelet on Sep 24, 2019, 20:12

Another issue is that I don't see how the geographical information fits in

What about relationships node__temporal_block and unit__node__direction__temporal_block? Is that what you mean? These are not included in the discussion to focus on the current issue, but are already part of Spine Model.

I fully agree with Manuel here, the geographical information is handled via the unit__node__direction__temporal_block relationship.

As we have discussed earlier, at some point it would be very convenient if the toolbox allows creating a lot of these relationships in a user-friendly way (however, that would I believe not be part of the database).

In GitLab by @manuelma on Sep 25, 2019, 14:29

In Manuel's proposal, the temporal scope can be defined either in the rolling or in the temporal_block objects. I think this can lead to confusion (as I think Juha pointed out): sometimes the temporal_block object defines the entire scope of the simulation, sometimes just a fraction of it. Also, this leads to more complex rules to handle the different situations (e.g., what happens when absolute temporal block objects are defined and a rolling object is defined so both having absolute start_datetime and end_datetime parameters).

So maybe just change that part, do it as in Maren's proposal. Say the scope of the simulation is defined in the (absolute, non rolling) temporal blocks, and drop parameters horizon_start... and horizon_end... from the rolling object. The rest of the proposal should keep working just the same.

In GitLab by @manuelma on Sep 25, 2019, 14:35

If we change my proposal as above, it looks like the only difference between Maren's proposal and mine is the way they define a resolution that is relative to the rolling window:

In Maren's, you use the rolling__temporal_block relationship.
In my proposal, you use temporal_blocks with relative start_date_time.

In GitLab by @manuelma on Sep 25, 2019, 20:37

rolling is explicitly initiated by the rolling__temporal_block relationship (in Maren's proposal)

Is the above correct? I thought rolling is initiated by the mere presence of a rolling object, and rolling__temporal_block serves to override the time_slice_duration of that block.

In GitLab by @jkiviluo on Sep 26, 2019, 07:20

First a general comment. I think it's really important to do this right. We will build new things on top of this. If we also do those things 'close enough', then in the long run we end up in a mess that's too hard to fix. I've been there and I would not like to go there again. So, I don't like the viewpoint that we already have something and that's why we have to do just incremental changes.

That said, I don't think my proposal is that much different especially when it comes to Spine Model - it's just how to interpret the structure. We would still use unit__node__direction__temporal_block in Spine Model - it just gets interpreted from the database based on which nodes unit converts energy between and what is the temporal resolution in those nodes.

Maybe I was trying too much at once, when I both proposed new names and somewhat different structure. I maybe should have kept the name temporal_block instead of time_section. However, I feel the names are important for comprehension by outsiders and we should trust the dictionary.

Comments and clarifications:

Proposal Juha: temporal scope of the simulation is represented via the rolling object (time_section in Juha's terminology)

No. There is the time_frame object that has the actual time frame to be modelled (e.g. year 2020). And the point is that it is then easy to change what year (or other time period) you want to model, because it's always in one and same place. I think this is really important and I hope this is understood before deciding on what to do.

Proposal Juha

I see some potential problems with this proposal:

Need 2 objects for the simple single-shot optimization

Not necessarily. As stated, I'm open to doing a shortcut for this (you could define everything in the time_frame for single-shots and then you don't need any time_sections)

As temporal_block objects (time_sections) are by default relative to the rolling object (time_frame), it is not clear to me how different representative periods would be modeled. As suggested, one option would be via the start_datetime/end_datetime parameters of the rolling object holding arrays. However, what if we would want to have different representative periods that use different resolution (or have a different resolution pattern within)? How to indicate which temporal_block object(s) corresponds to which representative period?

I suggested two alternatives for this, but time_frame is not a rolling object - it's the static definition of the time frame you want to model. So, in the static time_frame object you could state static representative periods through an array. Or you could define relative time_sections (e.g. relative to the beginning of the time_frame or to the beginning of a year) and attach those to a single node_group that contains the spatial scope of the whole model. With the latter approach you can do different resolutions etc.

it is unclear to me how the different temporal_block objects are related to each other (given that they do not rely on absolute temporal information). For instance, in the figure, there are 4 temporal_block objects: two of them have a relative parameter value of 1, and two have a relative parameter value of 2. How do we know which 2 follows which 1?

The 'relative' is not a parameter. It's an indicates the order - these are ordered sets. I know we don't support that yet, but it could be first done with a parameter value (spineInterface.jl would interpret) and replaced later with an ordered set. And the 2's don't get mixed up, since they are part of separate ordered sets.

unclear to me where the information regarding how to roll is stored?

Right, I forgot that. It could be in the time_frame - and it would be a natural place, since then you would control the high level stuff from one place. However, I'm not sure if there are use cases where different parts of the model would roll at different frequency. In this case, the 'jump' information should be in the optimization_window. Maybe both could hold it, but if both do so at once, then there should be an error.

every optimization is by default a rolling_horizon. This means the user can unintentionally start a rolling_horizon...

No. If you haven't defined a 'jump' then it won't roll. However, I think you should get a warning if your time_frame is full year but your optimization window is only two days and there is no jump defined.

In GitLab by @jkiviluo on Sep 26, 2019, 07:47

Another issue is that I don't see how the geographical information fits in

What about relationships node__temporal_block and unit__node__direction__temporal_block? Is that what you mean? These are not included in the discussion to focus on the current issue, but are already part of Spine Model.

Yes, but I don't think it's convenient to connect each node with temporal_blocks and especially all those unit__node__direction__temporal_block combos in the database. We should have a way to do this for the whole geography at once and that's what my proposal is about. Those combos can then be generated with the spineInterface.jl

In GitLab by @DillonJ on Sep 26, 2019, 08:23

@jkiviluo

Yes, but I don't think it's convenient to connect each node with temporal_blocks and especially all those unit__node__direction__temporal_block combos in the database.

We talked before about geographic aggregators, that are essentially arbitrary collections of nodes - you could link these to the temporal blocks.

However, I feel the names are important for comprehension by outsiders and we should trust the dictionary.

I don't see a problem with the current names. Also, the dictionary isn't that good at conveying the everyday use of words that have many meanings and the more likely understanding of these. For example "section". I don't like this word in the temporal context... to me it sounds too physical... like a pipe section or a section in an architectural drawing. For example, you would talk about code blocks more often than you would talk about code sections.

I think it's really important to do this right.

I absolutely agree and the ability to conveniently and flexibly model different geographical areas will be another key differentiator of Spine Model and something that is missing in other tools. However, we have the usual dilemma of either doing something good soon or doing something better at an unspecified future time. In my experience of the project so far, it takes a long time to get to a high level proposal that we're all comfortable with and longer still to get from there to something we can implement.

If we are to have any chance of having something implemented in time to finish the A case studies, I think we would need to see a proposal developed to the same level of detail as @mihlema's and @manuelma's

I think the key here is going with one of the two well developed proposals and having a good idea of how we can incorporate the geographical dimension easily - even if that means slight modifications. Again, to my point of relating node_goups (geographical aggregators) to temporal blocks...

In GitLab by @jkiviluo on Sep 26, 2019, 08:37

Just for clarity, here is what my proposal contains:

And what I think Maren's proposal contains:

So, in short, my proposal does two changes (which I don't believe are more difficult to implement): 1) it adds the geographical aggregators, which should be done for the other proposals too. We can implement them later also in my proposal, if we can live without them for now by just assuming that any rolling (optimization_window) object in the database gets automatically applied to the whole model. 2) It moves the absolute time definitions to one place. We can do also this without the model_instance and node_groups if we assume (for now) that the values in the time_frame will apply to the whole input database. I think it would be really cumbersome to start changing those dates in all of the temporal blocks when you want to model another year. Or if you want to change the length of the first time_section, then you need to change everyone after that too. And also the readability of those dates is much worse than just saying duration = x and resolution = y in a particular temporal_block.

In GitLab by @jkiviluo on Sep 26, 2019, 08:50

On blocks and sections. I think section is good exactly because it indicates order like sections in a pipe. These time sections are in linear order. Blocks are something that you can put together in various ways (2D or 3D). However, when we combine the time and the space, then we start to have blocks (those tempo-spatial blocks proposed a while ago).

Also, I think time_block or time_section is nicer than temporal_block or temporal_section. The temporal feels unnecessarily grand for this purpose. And time is shorter.

Rolling on the other hand feels bit vague. What would something named 'rolling' contain? On the other hand 'optimization_window' should be more clear for a newcomer.

In GitLab by @DillonJ on Sep 26, 2019, 09:11

On the wording - section still sounds wrong in this context. Like I said, you'd never talk about a section of code.. or a section of time. You would talk about a block of code or a block of time. Block implies contiguity for me. Section is a piece of something. The plumbing section in the hardware store...

optimization_window is probably better than just rolling. rolling_optimisation_window is very nice and clear but possibly too verbose. Rolling_window could work too.

We would need to see how your proposal can achieve the 5 use cases, spelled out to the same level of detail as the others with parameter values etc... but now we need new use cases to cover the geographical dimension.

@Juha In you proposal it looks like node_groups can have their own independent optimization_windows. What is linking them together? Are they independent optimizations? How does this work in the model? Are they solved sequentially? Put another way - in the other proposals each rolling object contains the full model... but in your proposal you have different parts of a model in different optimisation windows and it isn't clear to me how that can work?

In GitLab by @DillonJ on Sep 26, 2019, 09:11

closed

In GitLab by @DillonJ on Sep 26, 2019, 09:11

reopened

In GitLab by @DillonJ on Sep 26, 2019, 09:12

Sorry, hit comment and close accidentally!

In GitLab by @DillonJ on Sep 26, 2019, 09:14

Another point is that your proposal is 4 levels of hierarchy. I don't see what the highest level time_frame is adding... any parameters could just be added to model_instance instead. The fewer objects that a user has to create and relate to each other the better in my opinion.

In GitLab by @jkiviluo on Sep 26, 2019, 13:18

We would need to see how your proposal can achieve the 5 use cases, spelled out to the same level of detail as the others with parameter values etc... but now we need new use cases to cover the geographical dimension.

My proposal (https://gitlab.vtt.fi/spine/model/issues/120#note_10125) has the same level of detail as Manuels. Maren has more on hers, but at least for me it's easier to read the graph than the list. Maybe it requires some thinking on the part of the reader to see how the different cases can be organized, but they are basically there. I can do a list too, if really needed (I'm just terribly low on time).

@Juha In you proposal it looks like node_groups can have their own independent optimization_windows. What is linking them together?

I think we've covered this ground before. Roughly speaking (and those who know better can correct me), we decided that spineInterface.jl creates the t_overlaps_t, t_in_t, etc. based on the different resolutions that nodes have. Then flow_tuples(c,n,u,d,t) get generated based on that. If a unit converts energy between nodes in temporal_blocks of different resolution, then the summing of time steps must match (so that there is no undefined overlap as discussed some time ago - couldn't find the slides). This can be first left to the responsibility of the user, but we could have checks later that would find out if undefined behavior has been created. We will have this problem in all of the proposals once we want to have different resolutions in different node_groups.

Another point is that your proposal is 4 levels of hierarchy. I don't see what the highest level time_frame is adding... any parameters could just be added to model_instance instead. The fewer objects that a user has to create and relate to each other the better in my opinion.

Yes, you could have those parameters also in model_instance. However, having a separate object creates more flexibility in more advanced scenarios. You could have multiple time_frames to hold different years you might want to study (or just have a 2-week time_frame for testing and a 1-year time_frame for the real runs) or you could string time_frames together for e.g. myopic investment optimization case. On the other hand you could also want to change the model_instance to something different, e.g. between Spine Model and PowerSystems.jl, but use the same time_frame. Of course you can also just create multiple model_instances with separate time related settings or use alternatives on the time related values. I guess it's a matter of preference and also how convenient the relationship selection can be made.

In GitLab by @jkiviluo on Sep 26, 2019, 13:20

On the wording - section still sounds wrong in this context. Like I said, you'd never talk about a section of code.. or a section of time. You would talk about a block of code or a block of time. Block implies contiguity for me. Section is a piece of something. The plumbing section in the hardware store...

I can live with block. However, I would like to at least switch to time_block. As you write there 'block of time' is a thing. However, 'block of temporal' is not very clear to me.

In GitLab by @DillonJ on Sep 26, 2019, 13:29

...As you write there 'block of time' is a thing. However, 'block of temporal' is not very clear to me.

You're doing it wrong! Time is the noun, temporal is the adjective... you could write block_of_time if you like :-)

In GitLab by @manuelma on Sep 26, 2019, 14:32

Roughly speaking (and those who know better can correct me), we decided that spineInterface.jl creates the t_overlaps_t, t_in_t, etc...

Actually SpineModel.jl creates those sets now...

...based on the different resolutions that nodes have.

...based on the temporal_blocks. (Yes, some of these temporal_blocks might be specifying node resolution, through the node__temporal_block relationship, but that's another story.)

In GitLab by @manuelma on Sep 26, 2019, 15:02

My proposal (https://gitlab.vtt.fi/spine/model/issues/120#note_10125) has the same level of detail as Manuels

In my proposal I list the object classes involved, their parameters, and the type of each of these parameters. I also provide a section explaining how all this is interpreted by Spine Model where I cover all cases (How it works, generally speaking). I don't find this in your proposal @jkiviluo...? or maybe I'm missing something?

In GitLab by @jkiviluo on Sep 27, 2019, 07:53

Sorry about that. Indeed your proposal had more information. I've now added those same things to my proposal above. We still both lack the list of example data for each use case that Maren has.

I also have an additional use case 6 where the model can have a different resolution in different parts of the model.

In GitLab by @jkiviluo on Sep 27, 2019, 07:57

I added this to my proposal above:

We could also collapse the optimization_window and time_section into one optimization_window object by using arrays of duration and resolution in the optimization_window. E.g.

resolution = [3h, 6h]; duration = [24h, 72h]; or
resolution_for_duration = [(3h, 24h); (6h, 72h)]
the order would then come automatically

It might actually be better than separate time_section/temporal_block objects. It's probably more handy to use, but it's also bit more involved than separate time_section/temporal_block objects. Of course this depends on how nice interface we manage to create for the purpose (ordered sets).

spine-tools / SpineOpt.jl