Temporal Structure Proposal #22

Closed spine-o-bot closed 3 years ago

In GitLab by @DillonJ on Jun 22, 2018, 07:21

I have taken the liberty of creating an issue to discuss Juha's temporal structure proposal (attached).

Proposal_for_the_temporal_structure_of_the_Spine_Model.docx

[edit by Juha] There is a wiki page on the specific data structure that contains the temporal structure: https://gitlab.vtt.fi/spine/data/wikis/Spine-specific-data-structure-(for-Spine-Model)

In GitLab by @DillonJ on Jun 22, 2018, 07:27

I have attached a power point presentation outlining my idea to take what Juha has proposed and make it a little more generic. The idea is that we have a scheduling horizon object that will consist of an arbitrary number of related stages. We can define the timestep resolution, total duration and stochastic resolution for each stage independently. In the slides I give some examples of how it might work and how we might implement the structures proposed in Juha's document.

Additionally, the focus of Juha's document seems more to be around the stochastic model rather than the temporal structure (both of which are of course closely related).

Temporal_Structure.pptx

In GitLab by @jkiviluo on Jun 26, 2018, 20:14

That's pretty much as we have in Backbone. One difference is that in Backbone the stochastic structure has been separate from the 'stages' where the time step resolution is defined. Consequently the tree can branch out within a stage. However, it might be more clear to keep them both in the same scheduling horizon objects as Jody suggests.

In GitLab by @Poncelet on Jun 27, 2018, 10:46

I think the scheduling horizon object as presented by Jody (and Juha) could definitely be a starting point (certainly for operational problems).

However, there are a number of things which are not encompassed by the scheduling horizon object or need to be cleared out:

The multi-level part: in an investment planning model, one can have different years, each represented by a number of representative days/weeks/etc, which in turn are represented with a certain resolution. I believe we should be able to have different levels (as much as the user wants, defined in a flexible way). These levels would also relate to the tracking of different commodities (e.g., balance of coal is likely sufficient to track at an annual level), as described in issue #10 (Tracking commodities at different levels).
A way to describe how the variables should be generated for these different levels/stages.
- Ex 1: investment variables are only generated for the annual level (but how to specify?), whereas operational variables are to be generated at a lower level (possibly, see issue #10 again)
- Ex. 2: In a ED problem, UC variables of some units should be fixed, etc.

In GitLab by @DillonJ on Jun 28, 2018, 07:25

Over in the operational and investment problems discussion https://gitlab.vtt.fi/spine/model/issues/11 I was suggesting the following:

My temporal structure proposal is that we define a Scheduling Horizon object which can have a number of related Temporal Stage objects. E.g. we could have a single hourly deterministic temporal stage - or we could use this structure to define stages of different time resolutions etc...

We could then have on top of this an "Investment Model" object. We would then relate to this a Scheduling Horizon object. This allows us to define the investment problem in a very flexible way. The investment model object will hold all the parameters associated with the investment problem (e.g whether there is an investment loop with decomposition (investments as parameters in the scheduling model) or endogenous capacity variables directly in the scheduling model). The horizon object then has all the information about the underlying operational model.

In GitLab by @Poncelet on Jun 28, 2018, 07:45

@DillonJ : That sounds like a good idea.

However, on top of that, I would like to have similar levels within each year. This relates to Issue #10, where I think the 'commodity_timestep_multiple' is not sufficient because:

It creates a lot of variables which are not needed (which are easy to eliminate for the solver, but still appear in the final results, etc.
It is not flexible enough to say for instance that I want to track coal at an annual/seasonal level.

What I believe we need is to allow the user to define an hierarchical temporal structure in a generic way (for example, to have a year divided into seasons, which in turn are divided into week and weekend days, which in turn consist of a number of hours). If these different levels can then be referenced to, than we can simply say that the coal commodity is tracked at a certain level (and therefore, only variables at this temporal level will be generated).

I guess there are some similarities with the NodeGroup/GeoAggregator concept discussed in Issue #4 on the Geographical Model Structure.

In GitLab by @DillonJ on Jun 28, 2018, 08:56

@Poncelet I like this idea and agree it is needed

allow the user to define an hierarchical temporal structure in a generic way (for example, to have a year divided into seasons, which in turn are divided into week and weekend days, which in turn consist of a number of hours)

Maybe we could link with the time pattern feature here. We would have base time labels that we would use to define standard time units (e.g. D for day and H for hour and M for month etc.) We should be able to use these to define arbitrary time labels that we can then use in time_patterns.

For example, I should be able to define a time label "summer" that is defined as M6-M9. Or "night" that is H23,H1-H7

Or maybe Julia has similar functions and we need to link with Julia syntax to create a time label or pattern. This time pattern idea might need its own thread in the end - I will create one.

Back to the main point - this could be tricky, because the structure you are suggesting isn't truly hierarchical in that, for example, all seasons aren't necessarily of equal duration. You would need to define something like "quarters" for it to be truly hierarchical. Similarly for weekdays and weekend days. It's a very specific structure, and this is why I'm suggesting linking it somehow with the time_pattern idea.

Needs a little more thought. Any ideas @manuelma?

In GitLab by @jkiviluo on Jun 28, 2018, 10:16

I would agree that we need to enable the user to make a hierarchical (semi-hierarchical) temporal structure. It will be bit tricky to define well, but I think the ingredients should mostly be in the above discussion. Samples can be part of the solution too (it's probably important to have another dimension besides time steps in Spine Model to actually implement changing values without defining them for each time step).

In GitLab by @Poncelet on Jul 2, 2018, 09:36

@DillonJ : I like the idea of these 'labels'. I believe these are similar to node_group (geo-aggregators) in the geographical representation, so let's call these timestep_group? Having such timestep_groups would allow to define the temporal structure very flexibly. We could then use the time pattern idea to define these timestep_groups (although it should not necessarily be specified via time patterns).

What is still unclear to me is how to impose relationships between these different levels. See example in the figure below:

temporal_structure

How will we define that the timestep_groups 'Summer_Weekday' and 'Summer_Weekend' are on the same level in the temporal structure, and that these are both child timestep_groups of the timestep_group 'Summer'?

I'm thinking we need something as a 'Temporal_level' object which we could define, and reference to. There could then be a relationship between a timestep_group and a temporal_level object. In addition, we can use this to specify that a certain commodity is tracked at this temporal level (relationship between a commodity and a temporal_level object).

@DillonJ : I don't see why the temporal structure is not hierarchical? As I see it, it is not necessary (or more strongly, not desirable) that all timestep_groups (e.g., seasons) are of the same duration. Their duration should be determined based on:

The duration of their child timesteps/timestep groups
The number of times these child timesteps/timestep groups are assumed to be repeated within one occurrence of the parent timestep_group

In GitLab by @jkiviluo on Jul 3, 2018, 08:23

Not sure if this is possible, but I wonder if it would be preferable to use notation like temporal_group.summer.weekday.night

However, I don't see how these could be used in the core model. That would be hard-coding and hence they should stay in the data side and be translated to a Spine Model structure on the way. Conceivably these might still be used in user created constraints.

The whole thing is bit tricky, since we are considering dimensions.

One way to achieve different temporal levels in the core code is to use separate dimensions for selected variables. This is most clear between investment and operational variables like this (investments defined over samples and operational variables over time steps - these would naturally be linked so that one sample covers e.g. year of time steps):

v_invest(node, unit, sample)
v_flow(unit, forecast, time)
v_online(unit, forecast, time)

Furthermore, we can use the same index to achieve different temporal representations if we limit the set members (using GAMS like syntax as an example):

q_balance(node, forecast, time)$temporal_spatial_group(node, time)

Here the purpose of the temporal_spatial_group would be to allow different temporal representations in different nodes (or energy sectors). E.g. we could then have

gas.T001 = true
gas.T002 = false
gas.T003 = false
gas.T004 = true
...
elec.T001 = true
elec.T002 = true
elec.T003 = true
elec.T004 = true
... Dynamic variables will need some pre-calculations to get the right pointers, but that works ok in Backbone, although it is bit messy.

If we do this, there needs to be some translation where particular temporal_groups would be applied. They could be applied at least over nodes, node_groups, units, unit_groups, commodities, and commodity_groups. Hence, my example above is oversimplified and needs some additional structure before it can work.

In GitLab by @DillonJ on Jul 3, 2018, 08:56

Not sure if this is possible, but I wonder if it would be preferable to use notation like temporal_group.summer.weekday.night

@juha_k This is related to the time varying parameters discussion https://gitlab.vtt.fi/spine/model/issues/19 and similar to the idea of using time patterns and we need to link these two closely related topics.

The whole idea of time patterns is that the user can define arbitrary time groupings using a set of standard symbols (like Y for year or M for month and h for hour etc. etc.) E.g. the user can define a time label "Bedtime" as Bedtime=H23, NotBedTime=H1-H22,H24 or Lunchtime=H12-H13 etc, etc.

Then when defining a time varying parameter we can have something like this in the JSON field: value{ Timepattern[Bedtime,100],Timepattern[NotBedTime,200]}

But this is distinct to the present discussion, but related in that would should have consistency in the notation and perhaps have an element of commonality where we have a single mechanism to define custom time labels that can be used in time patterns or in defining the temporal structure.

In GitLab by @Poncelet on Dec 4, 2018, 11:15

I've made an update of the model description to include a proposal for the temporal representation on the model side.

A description of the proposal could be found in the Model-description repo, temporal_structure branch). All parts of the document related to the temporal structure, I've put in blue. This includes:

A new Section 1 describing the temporal structure
New versions of some constraints with this temporal structure. The included constraints are
- fix_ratio_out_in
- ramping constraints
- flow_bound
- cumulative_flow_bound
- nodal_balance_constraint

I myself am quite happy with this representation of temporal elements in the model. I think it provides a whole lot of flexibility.

In the document, I have focused on the model. What is not included in the document is:

how to represent this in the data structure
the link with the scheduler objects, time patterns, etc. (I considered these as wrappings around the core temporal model structure).

In GitLab by @Poncelet on Dec 13, 2018, 12:31

For those interested, I have made some of the equations more compact (cum_out_flow_bound, nodal_balance) based on our discussion yesterday -> to be seen at Model-description repo, temporal_structure branch).

In GitLab by @manuelma on Dec 13, 2018, 14:13

I'm reading that you managed to compound the two sums into one using 'time_slice_path'? That's great.

In GitLab by @DillonJ on Dec 17, 2018, 10:51

Hi all...

I have had a stab at merging Kris' proposal with the original ideas for the looping structure, into a single unified proposal. I have created a wiki page here: https://gitlab.vtt.fi/spine/model/wikis/Temporal-and-Looping-Structure

We probably need another call to discuss, but I feel that we should move forward too fast with the single model instance temporal structure until we have a basic idea of how we're going to do the looping etc, because they are closely related.

Again, the time patterns discussion is separate, as this is just a mechanism to specify time varying data in the model. We may use some of the functions to define time intervals, but they are different things and independent to a large extent.

In GitLab by @jkiviluo on Dec 21, 2018, 18:13

Looks very good. But I have some comments and suggestions.

This is a very fundamental structure and if it is not flexible enough, we will preclude possible ways to model. Hence, we should not restrict the structural definition to temporal stages only. Obviously Commodity dimension can be included using the commodity_stage_multiple so that different commodities can have a different time resolutions, but the line will not always be between commodities. The line could also be between nodes (e.g. GB and Ireland) or even between units. Furthermore, commodity_stage_multiple is not sufficient to enable different forecast structures between geographical entities. I would therefore suggest that instead of just 'temporal stages' we would also have a higher level entity 'spatio-temporal blocks' that combines temporal_stage and geography (understood more widely so that it includes nodes and units). The user can then define the temporal structure in two dimensions: time and geography. The temporal structure of the whole model is then constructed from these blocks automatically by JumpAllOut - the seams will require correct pointers and JumpAllOut would manage that.

A block could be constructed by making an 'uberGroup' over unit, node, connection and possibly storage objects to which a one or more temporal stages could be related. And in order to keep things simple for simple models, there would be an automatic group with all geographic objects, which can then be related to the default temporal structure automatically (and that default temporal structure can come from an instance of a model Archetype).

Consequently, optimisation horizon might also vary between spatio-temporal blocks - maybe it should just be an organic result from the blocks that are combined to form the whole model and there is no need for separate optimisation horizon object.

This all of course links to the hieararchical time structure Kris has presented. I think there is two separate issues: 1) Clearly, the hierarchy is important when defining time dependent data: sometimes it's important to use hourly values while other times monthly might be enough. This should of course be allowed, but it's a separate question from the temporal structure of the model. Even if some prices were monthly, we might still want to model at hourly level (and use flat price for that particular commodity). I believe the way JSON has now been developed (Manuel made the latest iteration) would fully support data entry at different temporal resolutions. 2) Another viewpoint on that hierarchy structure is that sometimes we do want to model different things (e.g. commodities) at different time resolutions. The block approach would fully allow this - you can just think the 2nd dimension of the blocks (geography) to be the hierarchy dimension in Kris's figures. And if this doesn't sink in, we should probably try to work through an example.

The proposed looping structure with loop levels looks promising, but we will probably add detail once we need to actually use it.

I understand we need a fundamental_time_interval, but I'm not sure if it's better to have a separate object_class for it. It could also just comes from connecting time_interval object to the model instance.

In GitLab by @Poncelet on Jan 3, 2019, 09:57

Thanks for the stab Jody!

I'm not sure whether I'm fully on board with some of the ideas presented here. The wiki mainly describes things within one optimization horizon. However, these are things which are already represented (somewhat differently) in Spine model side. We have:

time slices which can have a different duration
a way of defining model side how frequently to do nodal balances (combining commodity and geography)
a way of defining for which time slices to define the flow variable

In this setting, I'm not sure how to interpret the ideas like the temporal stage, and the temporal interval. Are these: a) things we want to use data side to represent the temporal structure more conveniently (and then translate in Jump_all_out or Spine model to the data structure used in Spine model? OR b) an alternative proposal for the temporal structure?

Assuming it is case a), I'm worried that the distance between data side and model side become quite big (with the temporal stages etc., there seems to be some kind of mapping from those temporal stages to the actual temporal structure). Why don't we directly represent the temporal structure in the data as in Spine model? I also believe it will be difficult to achieve the same level of flexibility with this higher level representation via temporal stages etc. in between.

So what for me still are open issues are:

How to define the temporal structure of a single optimization horizon in the data (assuming model side remains untouched): direct representation or have some higher level things like temporal stages etc.?
How to define the looping structure (multiple optimizations done sequentially)

Regarding the hierarchical structure:

It is not clear to me how to represent the hierarchical temporal structure in your proposal?

Clearly, the hierarchy is important when defining time dependent data: sometimes it's important to use hourly values while other times monthly might be enough. This should of course be allowed, but it's a separate question from the temporal structure of the model. Even if some prices were monthly, we might still want to model at hourly level (and use flat price for that particular commodity).

I don't see how this is a separate question from the temporal structure of the model. If we want in some case to have flow variables on a monthly level, there should be a way to specify this, and that is part of the temporal structure of the model?

Another viewpoint on that hierarchy structure is that sometimes we do want to model different things (e.g. commodities) at different time resolutions. The block approach would fully allow this - you can just think the 2nd dimension of the blocks (geography) to be the hierarchy dimension in Kris's figures. And if this doesn't sink in, we should probably try to work through an example.

This I do not fully understand?

Can we have a call today or tomorrow to discuss this?

In GitLab by @manuelma on Jan 3, 2019, 14:38

I'm up for a call!

In GitLab by @jkiviluo on Jan 7, 2019, 14:19

I am available for discussion this week almost at any reasonable time.

I should probably make a more clear presentation of what I proposed three comments up (especially 2nd issue at the end of the message). It tries to join together what Jody presents up there and how Kris and Maren have approached the temporal structure so far. Plus it adds some things that I feel are necessary for full model adaptability.

When it comes to what's in the Toolbox data structure and what is in the Spine Model, I would first say that the temporal structure in the Toolbox needs to be comprehensive but easy to modify. It should allow data input in different resolutions while still enabling the modeller to make different selections for the modelling resolution (in time and space). However, Spine model should keep it simple so that equations do not become unwieldy. It means that there needs to be a translation in Jump_all_out. However, Spine Model needs to get a looping structures and such things too (so that it does not need to communicate with Toolbox between solves). It's just that in Spine Model model structure should be presented in a manner that allows easily readable/writable equations. If the translation happens in the equations, then we will be doing it all over the place (probably using lot of conditionals). Julia is more flexible language, so it might give us tools to hide some of the complexity, but I would still be careful not to have too complex data structures in the Model side.

In GitLab by @manuelma on Jan 7, 2019, 18:14

We could also take some time to look at https://gitlab.vtt.fi/spine/toolbox/wikis/spine-toolbox-json-specification/generic-proposal and see where it fits, and if it doesn't overlap with already covered functionality.

In GitLab by @Poncelet on Jan 8, 2019, 10:45

I would first say that the temporal structure in the Toolbox needs to be comprehensive but easy to modify. It should allow data input in different resolutions while still enabling the modeller to make different selections for the modelling resolution (in time and space).

I believe I see two distinct elements here:

The representation of the fundamental temporal structure: how is this defined in Spine toolbox (data) side? Via things like Jody's temporal stages, temporal intervals, etc.; or directly have the time slices as objects (~Spine model formulation)
Handling of input data that comes in different resolutions than the temporal structure necessarily has.

However, Spine model should keep it simple Spine Model model structure should be presented in a manner that allows easily readable/writable equations

I believe Spine model is currently quit readable? Of course, at the moment there is no looping structure etc.

It means that there needs to be a translation in Jump_all_out.

If there needs to be a translation, I don't think it should be part of Jump_all_out. My thoughts on this would rather be that we have Jump_all_out to make all the object classes etc. directly accessible in Spine Model. Or any Julia model for that sake. I see Jump_all_out as part of the toolbox, or a kind of api between the toolbox and Julia models. If a translation is needed, this would be truly be a part of Spine model, and therefore better in another script I believe. What I believe this script should do is to create/fill/modify the sets used in Spine model (e.g., set of time slices, set of temporal levels), the associated parameters (e.g., weight and duration of the time slices), and to make sure that parameter values can be accessed properly (e.g., averaging if data at a higher resolution).

In this regard, what I think we should try to get clear is what information we store in the toolbox (and pass to Jump_all_out).

Summarized in a schematic :)

Spine_temporal

In GitLab by @jkiviluo on Jan 9, 2019, 09:17

That's cool. The temporal script could be in Jump_all_out, in Spine Model or as another separate script. If it's in Spine Model, then it's not usable for other models/tools that might want to use it too. But it may of course be too specific for other tools. It might be best to develop it as separate and see where it goes later.

Spine Model is readable now, but I'm worried what happens when we add the stochastic structure, the capability to have different temporal resolutions in different parts of the model, and the looping structure. And the current equations are not the most difficult ones - it's the dynamic ones like start-ups with hot/warm/cold that can become nightmares.

In GitLab by @jkiviluo on Jan 9, 2019, 09:21

I've made some slides to better explain the tempospatial blocks I proposed earlier https://docs.google.com/presentation/d/1mwGX6mm3PwbhJ9OvLlnS5Vm17y1HpPdaiouCrFEGonk. I can go through them shortly in the telco.

And I think we should also shortly discuss the JSON Manuel was referring to. I think it's fine and allows us to do what we want (in any way we might choose).

In GitLab by @Poncelet on Jan 9, 2019, 09:41

Spine Model is readable now, but I'm worried what happens when we add the stochastic structure, the capability to have different temporal resolutions in different parts of the model, and the looping structure.

The capability to have different temporal resolution in different parts of the model is already integrated in most of the model description, so I think that is ok. Regarding the looping structure, I see it as something outside of the individual equations, and hence not impact the readability of the equations. The stochastic structure will complicate things a bit more, but I believe that is still feasible.

And the current equations are not the most difficult ones - it's the dynamic ones like start-ups with hot/warm/cold that can become nightmares.

Yes, that is something we still need to have a look at (probably in one of the case studies).

In GitLab by @DillonJ on Jan 9, 2019, 09:55

A quick thought... I'm not sure why you need to define tempo-spatial blocks... you can just define temporal blocks, or stages, and relate them to geographic areas to implicitly create the tempo-spatial blocks. So we can achieve what you want within the EAV structure that we already have.

I think we also don't want to fall into the trap of defining the what and the how all at the same time. We certainly do want to have some ideas about the implications of our proposals, but we shouldn't be too prescriptive.

I think we should try and refine our high level ideas with an EAV mindset without delving too deep into the how... otherwise, we just get bogged down.

In GitLab by @DillonJ on Jan 9, 2019, 09:59

@Poncelet One comment on your post above is that our concept of parameter is quite generic even at the database level... so there is just a parameter.. and a parameter can be a time series, can be time patterned or can be static and JumpAllOut will handle this. So I don't think we need to make special distinctions between time series parameters and static parameters. JumpAllOut already handles all this

In GitLab by @Poncelet on Jan 9, 2019, 15:38

One comment on your post above is that our concept of parameter is quite generic even at the database level... so there is just a parameter.. and a parameter can be a time series, can be time patterned or can be static and JumpAllOut will handle this. So I don't think we need to make special distinctions between time series parameters and static parameters. JumpAllOut already handles all this

Fully agree. I just separated it too point out that there is work to be done on the exact JSON schema to use.

In GitLab by @jkiviluo on Jan 11, 2019, 10:17

Here's the simple conversion equation using flat time mapping. p_coeff is assumed to be available for the lowest t so that the equation can be more simple (Julia can handle that). I realize Kris has a co-efficient (ratio) only in the input commodity side, but I haven't yet understood how that would work in situations where there are two output commodities with different co-efficients.

In GitLab by @Poncelet on Jan 11, 2019, 10:40

I realize Kris has a co-efficient (ratio) only in the input commodity side, but I haven't yet understood how that would work in situations where there are two output commodities with different co-efficients.

To clarify, I have a parameter ratio_out_in which has a double commodity (actually commodity group) index, so e.g. with two output commodities,

fix_ratio_out_in(Electricity, Coal) = 0.40 => equation will be v_flow(Electricity) = v_flow(Coal) * 0.40
- fix_ratio_out_in(CO2, Coal) = 3 (arbitrary number)0000 => v_flow(CO2) = v_flow(Coal) * 3

In GitLab by @jkiviluo on Jan 11, 2019, 10:45

But how it would work when you sum two outputs, e.g. heat*1.3 + elec >= 100

...but we shouldn't derail there in this issue! We can use slack to discuss that if needed.

In GitLab by @Poncelet on Jan 11, 2019, 11:19

Let's continue that discussion on slack indeed.

More important: from a temporal perspective, I see that your equation is almost identical except for the additional row in my version.

I have this additional row in my equation for the situation that the parameter fix_ratio_out_in/p_coeff would be defined at a higher resolution than some of the flow variables.

For instance: coal flow is at daily resolution, but for some reason the efficiency (p_fix_ratio/p_coeff) varies from hour to hour.

In your case, you take the daily flow coal (time slice t'), multiply it by the duration of t' and then multiply it be p_coeff(t) -> but what is this t value?

In my case I take the time-weighted average value of p_fix_ratio/p_coeff and do this averaging operation directly in the equation (last row). What I think you implicitly suggest by saying "p_coeff is assumed to be available for the lowest t" is that Julia (but that is part of the model) handles that such that we can reference p_fix_ratio at the time slices t' for which the flow variables are defined (even though the fix_ratio values are possible entered by the user at a different resolution). So basically, the averaging operation that might be needed would be done outside of the equation.

If that is the case, I agree that can help to improve the readability of the constraint considerably (I'm making some changes to improve the readability of some other equations in a similar reasoning). But outside that, I think you have the exact same thing as I have I think (which is positive I guess :)).

Based on this equation, I do not yet see a difference between lateral and hierarchical time slice structure.

In GitLab by @jkiviluo on Jan 11, 2019, 11:55

I think the main differences are 1) in your version there is one more term (I assume that doesn't disappear by defining p_fix_ratio to the lowest resolution). 2) you have three time sets: t, t' and t1, whereas my version has two 3) understanding the sums for a hierarchical tree is more difficult than understanding the sum for the t_t' tuple (one has to keep in mind which level t, t' and t1 point to when reading the equation)

I don't see any particular benefit in the hierarchical tree and it might have further trouble to handle the situations where the temporal levels are partially overlapping (as discussed in the call).

In my view it would be wise to keep the time structure as simple as feasible. That equation is a piece of cake, but it was already hard to read. We have those start-up equations coming and also piece-wise efficiencies to be written into the conversion equation.

How Julia would handle the calculation of p_coeff I don't exactly know. jump_all_out is one option, but we shouldn't waste memory by populating it with the same number multiple times when not really needed. So, probably better to find a way to overload the parameter in the equation. Anyway, this doesn't need to show up in the written equation descriptions and I'm glad you're adopting that.

In GitLab by @Poncelet on Jan 11, 2019, 15:54

in your version there is one more term (I assume that doesn't disappear by defining p_fix_ratio to the lowest resolution). you have three time sets: t, t' and t1, whereas my version has two

Both the additional term and the additional index would disappear if p_fix_ratio is is calculated externally. I was planning to show you some hours ago, when I stumbled upon a problem in the current equation (both our versions) that I have not resolved yet. I'll come back to that.

understanding the sums for a hierarchical tree is more difficult than understanding the sum for the t_t' tuple (one has to keep in mind which level t, t' and t1 point to when reading the equation.

This I still doubt. Could you exactly define the meaning of the t_t' tuple? Is it:

t_t' have some overlap (similar to my time_slice_path(t,t')
t' is a part of t (similar to my time_slice_tree(t,t') In any case, it is something I have also used I believe. In that case, it is maybe more a matter of the naming/defining/visualizing then?

In any case, I tend to realize that in the end, both versions are quite similar in a lot of aspects.

I don't see any particular benefit in the hierarchical tree and it might have further trouble to handle the situations where the temporal levels are partially overlapping (as discussed in the call)

Thinking about benefits of hierarchy, and exploring these overlapping things are still on my agenda.

One question on the overlaps: I understand that we want to be as flexible as possible and that there should be benefits to the hierarchical when we would go for that. But assuming that these overlaps would not be possible, how big of an issue is that? Maybe more generally, let's say that the readability could be improved significantly without having the option to do these overlaps, what would we do? I have the feeling that it might be a fictitious issue (i.e., rarely encountered and easily solved by splitting up as Jody proposed)?

In GitLab by @Poncelet on Jan 14, 2019, 09:58

Update: I have adapted the fix_ratio_out_in constraint by handling the resolution of the parameter outside the equation. It now looks as follows (more or less identical to Juha's version (at least in time):

fix_ratio_out_in_v2

In GitLab by @jkiviluo on Jan 14, 2019, 12:03

Very nice. It looks similar indeed. I think, on the model side, we just need to discuss the terminology. I haven't used 'slice' before in this context, but the dictionary tells me that it's a pretty good name. Probably better than 'interval', 'step' or 'period' that we've been using.

'slice' would then say that is a sub-set of arbitrary number of fundamental time steps. We would call it t (or t') for short in the equations.

I'm more uncomfortable with the 'timeslice_tree'. In here it's just a mapping from t to t'. Hence, I'd rather call it 't_t_tuple' or 't_t_map' or something along those lines.

Then there is the question how this part of the temporal structure is actually set in the Data Store. Probably better to have a call to discuss that.

In GitLab by @Poncelet on Jan 14, 2019, 13:56

I'm more uncomfortable with the 'timeslice_tree'. In here it's just a mapping from t to t'. Hence, I'd rather call it 't_t_tuple' or 't_t_map' or something along those lines.

The naming we can definitely discuss. These I have decided rather quickly. What I would like to achieve with the names is that they are almost self-explanatory. I realize the names I've used are not succeeding in doing that, but I also think t_t_tuple, or t_t_map don't explain too much. The most frequently used sets in my version of the equations are

A set of tuples (t,t') with the meaning that for a tuple (t,t') in this set, there is some overlap between t and t'. -> possible names for this could be t_t_overlap, t_t_intersect, timeslice_overlap, ... Happy to get other ideas
A set of tuples (t,t') with the meaning that for a tuple (t,t') in this set, there is some overlap between t and t' AND t' has a higher resolution (smaller or equal duration) than t. -> I cannot directly think of good names for this.

I've made some slides to better explain the tempospatial blocks I proposed earlier https://docs.google.com/presentation/d/1mwGX6mm3PwbhJ9OvLlnS5Vm17y1HpPdaiouCrFEGonk.

I have added slide slides 5 and 6 to it to show how the information in the tempo-spatial blocks could translate to the hierarchical temporal structure. In those slides, I have assumed that the tempo-spatial blocks would be the toolbox side of representing certain information. I don't see a problem in representing these overlaps?

Then there is the question how this part of the temporal structure is actually set in the Data Store. Probably better to have a call to discuss that.

I'm happy to set up a call, but I don't know if we have advanced a lot on this front at this point.

In GitLab by @manuelma on Jan 14, 2019, 14:48

Then there is the question how this part of the temporal structure is actually set in the Data Store. Probably better to have a call to discuss that.

In my opinion, the best would be to precisely list what are the requirements for SpineModel in terms of sets, parameters and so on, and then start thinking how to generate those from data with JuMP_all_out.

In GitLab by @Poncelet on Jan 14, 2019, 15:06

I agree with that. Therefore, I think it is key to resolve the lateral versus hierarchical temporal structure discussion (I'm working on getting an overview of different possible benefits and drawbacks of both systems at the moment, but need some more time to think things through).

In GitLab by @manuelma on Jan 22, 2019, 21:46

I'm working on the data-side 'implementation' of the temporal structure described by @Poncelet. The partial result is here if you want to check it out (will appreciate any comments to know if I'm in the right direction.)

Two comments:

we still need to figure out how to incorporate the infamous looping structure into @Poncelet's temporal structure. One can think of the temporal structure as everything below and including @DillonJ's concept of optimisation horizon, and of the looping structure as everything above. However, the concept of repetition (similar to looping) is somehow present in the temporal structure through the weight parameter. I guess my point is the two are very similar, so we may wanna use a similar structure (temporal levels ~ looping levels) to represent both. Temporal levels belong in a single solve statement, whereas looping levels define each of them an individual solve.
@juha_k 's concept of tempo-spatial blocks. Are they already covered by the current temporal structure formulation, or we need to do something about it?

In GitLab by @manuelma on Jan 23, 2019, 14:58

In https://gitlab.vtt.fi/spine/model/issues/17#note_5352 there's the idea of having a 'separate script' in SpineModel.jl that creates sets and so on for using the temporal structure. I think we can have that script in the data itself, by incorporating expressions in the json field of parameter values. JuMP_all_out can parse and evaluate whatever julia expression we pass in the json, so this should work.

What do you think? Do you like this approach?

In GitLab by @Poncelet on Jan 24, 2019, 08:52

Nice work Manuel, I think this can work!

Clarifying questions:

The main motivation to enter the time steps as a parameter rather than an object is to avoid the multitude of relationships and make it easier for a user to enter all the time steps?
What does the parameter child_time_step exactly contain as values (what would a call (child_step() in Julia return after Jump_all_out? I think I understand that the expression uses the information specified in children_count and the temporal_level__child_level relationship to link the time steps in different temporal levels
Which steps would the user need to take when starting a new Spine model?

1): for defining the time steps in each temporal level, I assume this can be done by entering a list directly, but an expression would also be possible?

2): For Req.3, this all object and the json expression value for the time step parameter of this object, would that be part of an empty Spine database?

3): The children_count parameter would be set manually by the user I assume?

4) The relationships of class temporal_level__child level would need to be constructed by the user I assume?

5) The expression in child_time_step I assume to be also part of an empty Spine database?

6) The duration and weight parameters I assume can be also entered directly as a list?

Other thoughts:

up to now, I have not yet encountered the need for having this child_step parameter in Spine model. I think the descendant_time_step is sufficient
on top of the descendant time_step, what would also be nice is to have a parameter which gives a set of time steps in any level (higher, lower or equal) which overlap with a given time step in temporal_level
Similar to the link between time steps in different levels, we need a link between consecutive time steps for the dynamic constraints (the t_t_succeeds idea). I guess we can also derive a parameter with a corresponding expression for that?

In GitLab by @Poncelet on Jan 24, 2019, 08:58

we still need to figure out how to incorporate the infamous looping structure into @Poncelet's temporal structure. One can think of the temporal structure as everything below and including @DillonJ's concept of optimisation horizon, and of the looping structure as everything above. However, the concept of repetition (similar to looping) is somehow present in the temporal structure through the weight parameter. I guess my point is the two are very similar, so we may wanna use a similar structure (temporal levels ~ looping levels) to represent both. Temporal levels belong in a single solve statement, whereas looping levels define each of them an individual solve.

Possibly, but it is important to note that the temporal structure can be redefined within the looping structure. For instance, we do a loop where each solve has a time horizon of 36 years (6 hours initial condition, 24 hours actual solution + 6 hours for accounting for what comes after). During the first iteration in the loop, there are typically no starting conditions -> only do an optimization of 30 hours. In the last iteration, there is typically no period for accounting what comes after + the total time horizon that needs to be covered is not necessarily an integer multiple of the single optimization horizon.

In GitLab by @Poncelet on Jan 24, 2019, 08:59

@jkiviluo 's concept of tempo-spatial blocks. Are they already covered by the current temporal structure formulation, or we need to do something about it

I think this tempo-spatial blocks was an alternative way of specifying the temporal structure.

In GitLab by @jkiviluo on Jan 24, 2019, 09:38

I have similar questions as Kris. The JSON expressions are bit complex and a regular user should not need to worry about them. The user should just define temporal levels like 1h and 3h and state that the connecting tuple is needed. If they are not divisible (e.g. 2h and 3h), then the wish is not granted and an error message is issued instead.

As a side note, the word 'hierarchy' is not really needed. There can be arbitrary levels that are linked as required (e.g. 1h and 3h as well as 1h and 4h for other purpose within the same model).

I think this tempo-spatial blocks was an alternative way of specifying the temporal structure.

There is something more to it. We need to be able to say which temporal structures apply to which groups of nodes. Maybe it's just a relationship between temporal structure and nodeGroup. However, the system needs to be able to resolve the links between time, forecasts and locations so that it can generate the tuples needed. (As I've tried to express in the slides with 'tempospatial blocks')

In GitLab by @Poncelet on Jan 24, 2019, 12:24

There is something more to it. We need to be able to say which temporal structures apply to which groups of nodes. Maybe it's just a relationship between temporal structure and nodeGroup. However, the system needs to be able to resolve the links between time, forecasts and locations so that it can generate the tuples needed. (As I've tried to express in the slides with 'tempospatial blocks')

Yes, that was also something I forgot in my previous comment. What we also need is to say

for which time steps to do a market clearing for a node -> nodal_flow_balance_time_slice in the slides and model description
for which time steps to define the operational variables (for each flow of each unit) -> flow_tuples in the slides and model description.

I'm not sure how easy it is to do it. What would be simple is to establish a relationship between a node and a temporal level object, but this is not sufficient, as the resolution can change over time (and hence the temporal level). The straightforward way would be to have a relationship between a node and a time_step but given that these time_steps are not objects (for other reasons), it might be more tricky.

In GitLab by @Poncelet on Jan 24, 2019, 12:29

As a side note, the word 'hierarchy' is not really needed. There can be arbitrary levels that are linked as required (e.g. 1h and 3h as well as 1h and 4h for other purpose within the same model).

On this: with the temporal structure we cannot support having 1h, 3h and 4h time steps (even though 3h and 4h time steps would not come together anywhere in the model, it will be very hard to check and guarantee that a priori). If we would want to support this, this makes writing the equations more difficult (less readable), less efficient (creation of duplicates of the same equation and slower generation of constraints), as discussed in our call last week.

In GitLab by @manuelma on Jan 24, 2019, 12:51

thanks @Poncelet for the input. Here are my answers to your questions:

The main motivation to enter the time steps as a parameter rather than an object is to avoid the multitude of relationships and make it easier for a user to enter all the time steps?

Exactly. The alternative is to have a class called time_step where the objects are individual time steps --that would be more logical from the EAV point of view, but totally cumbersome in my opinion.

What does the parameter child_time_step exactly contain as values (what would a call (child_step() in Julia return after Jump_all_out? I think I understand that the expression uses the information specified in children_count and the temporal_level__child_level relationship to link the time steps in different temporal levels

If I call child_time_step(temporal_level=:day, t=2) I'd get something like [25, 26, 27, ..., 48] (the hours in the second day). But child_time_step shouldn't be used on its own. It's just there to simplify the expression for descendant_time_step.

Which steps would the user need to take when starting a new Spine model?

Basically define temporal_level objects, temporal_level__child_level relationships, and specify parameters time_step, children_count, duration and weight for each temporal_level object. My idea is that everything else (such as the json for descendant_time_step) comes with the 'empty' Spine Model database, so the user doesn't need to worry about it.

for defining the time steps in each temporal level, I assume this can be done by entering a list directly, but an expression would also be possible?

It could be a julia expression such as vcat(collect(1:12), collect(19:24)), which returns [1, 2, ..., 12, 19, 20, ..., 24]. Is that what you mean?

For Req.3, this all object and the json expression value for the time step parameter of this object, would that be part of an empty Spine database?

The children_count parameter would be set manually by the user I assume?

The relationships of class temporal_level__child level would need to be constructed by the user I assume?

The expression in child_time_step I assume to be also part of an empty Spine database?

The duration and weight parameters I assume can be also entered directly as a list?

Yes, all of this.

up to now, I have not yet encountered the need for having this child_step parameter in Spine model. I think the descendant_time_step is sufficient

Agree, child_time_step is just there to simplify the expression for descendant_time_step.

on top of the descendant time_step, what would also be nice is to have a parameter which gives a set of time steps in any level (higher, lower or equal) which overlap with a given time step in temporal_level

Similar to the link between time steps in different levels, we need a link between consecutive time steps for the dynamic constraints (the t_t_succeeds idea). I guess we can also derive a parameter with a corresponding expression for that?

These two requirements are present in your PDF but haven't been addressed yet. I need to keep working...

In GitLab by @Poncelet on Jan 24, 2019, 12:58

It could be a julia expression such as vcat(collect(1:12), collect(19:24)), which returns [1, 2, ..., 12, 19, 20, ..., 24]. Is that what you mean?

I meant more like directly entering [1, 2, ..., 12, 19, 20, ..., 24] but I am pretty sure that that should be ok :)

In GitLab by @manuelma on Jan 24, 2019, 13:21

Thanks @juha_k for the input. Here is my return:

I have similar questions as Kris.

I have answered Kris's questions above.

The user should just define temporal levels like 1h and 3h and state that the connecting tuple is needed. If they are not divisible (e.g. 2h and 3h), then the wish is not granted and an error message is issued instead.

In my proposal, the user would define two temporal_level objects, say 1h and 3h, together with a temporal_level__child_level relationship from 3h to 1h. Now, if they specify the duration parameter for each of these two temporal_level objects, we could figure out the children_count of the 3h level by dividing the durations as you suggest. I doubt this approach is correct, and that's why I make it so the user needs to introduce the children_count themselves. My main reason is the duration is not always the same for all time steps in a temporal_level, it can vary, so dividing becomes a nightmare.

There can be arbitrary levels that are linked as required (e.g. 1h and 3h as well as 1h and 4h for other purpose within the same model).

I didn't find this requirement in Kris's PDF? This may require rethinking the whole thing.

We need to be able to say which temporal structures apply to which groups of nodes. Maybe it's just a relationship between temporal structure and nodeGroup. However, the system needs to be able to resolve the links between time, forecasts and locations so that it can generate the tuples needed

Which are the tuples needed? Is something more than flow_tuple and nodal_flow_balance_time_slice from Kris's PDF?

In GitLab by @manuelma on Jan 24, 2019, 13:33

What would be simple is to establish a relationship between a node and a temporal level object, but this is not sufficient, as the resolution can change over time (and hence the temporal level).

@Poncelet I thought the resolution could vary within a single temporal level? Say, we define the duration parameter of the temporal_level object as something like [1, 1, 1, ..., 1, 3, 3, ..., 3] (if you know what I mean). So the relationship between the node and the temporal_level should work for the above purpose?

In GitLab by @jkiviluo on Jan 24, 2019, 14:09

There can be arbitrary levels that are linked as required (e.g. 1h and 3h as well as 1h and 4h for other purpose within the same model).

@manuelma: I didn't find this requirement in Kris's PDF? This may require rethinking the whole thing.

and

@Poncelet: On this: with the temporal structure we cannot support having 1h, 3h and 4h time steps (even though 3h and 4h time steps would not come together anywhere in the model, it will be very hard to check and guarantee that a priori). If we would want to support this, this makes writing the equations more difficult (less readable), less efficient (creation of duplicates of the same equation and slower generation of constraints), as discussed in our call last week.

It shouldn't need any re-thinking nor extra support in the model code. It should just work as long as 1h-3h and 1h-4h meet only through 1h and do not interact directly. We can add checks later if needed, but I would just see this as an advanced way to use the model where the user should be aware what she/he is doing. What I just meant to say that it's not really hierarchical tree any more and therefore the term becomes incorrect when strictly speaking.

spine-tools / SpineOpt.jl

Temporal Structure Proposal #22