Temporal Structure Proposal #22

Closed spine-o-bot closed 3 years ago

In GitLab by @DillonJ on Jun 22, 2018, 07:21

I have taken the liberty of creating an issue to discuss Juha's temporal structure proposal (attached).

Proposal_for_the_temporal_structure_of_the_Spine_Model.docx

[edit by Juha] There is a wiki page on the specific data structure that contains the temporal structure: https://gitlab.vtt.fi/spine/data/wikis/Spine-specific-data-structure-(for-Spine-Model)

In GitLab by @manuelma on Jan 24, 2019, 14:59

Maybe we can talk of parallel hierarchies, e.g., 3h -> 1h and 4h -> 1h, which are still hierarchies on their own.

But I have a question. Let's say we define these two parallel hierarchies. That shouldn't be a problem for determining the set time_slice_tree in @Poncelet's PDF (more or less equivalent to the parameter descendant_time_step in my wiki), since there's always a unique way down.

But what about the set time_slice_path, which goes both up and down? What should go there? Do we need two different time_slice_path sets (one for the 3h hierarchy and one for 4h hierarchy)?

In GitLab by @DillonJ on Jan 24, 2019, 15:13

@manuelma

Maybe we can talk of parallel hierarchies,

This is exactly the idea I was suggesting during the previous call. Instead I used the word "lateral" which perhaps gave the wrong idea, but parallel hierarchies is a better way of putting it.

I think this makes a lot of sense.

In GitLab by @manuelma on Jan 24, 2019, 15:32

So these parallel hierarchies or timelines are not supposed to interact with each other? How does the model handle this? Does it generate separate models (I mean solve statements) for each of them? Are results passed from one to another? Isn't this getting into the looping structure territory?

I guess my point is I didn't see any of this in @Poncelet's pdf, maybe I just missed it?

In GitLab by @manuelma on Jan 24, 2019, 18:22

I meant more like directly entering [1, 2, ..., 12, 19, 20, ..., 24] but I am pretty sure that that should be ok :)

In julia, [1:3..., 7:9...] is the same as [1, 2, 3, 7, 8, 9], so users could enter an expression like that. I like it a lot, think it's pretty simple.

In GitLab by @Poncelet on Jan 25, 2019, 08:24

On the parallel hierarchy: This would signifcantly impact the model formulation for following reasons:

Regularly the equations need to find the time slices which have some time in common (called time_slice_path(t,t') in the documentation (t_t_overlap(t,t') in the slides). As Manuel correctly stated, there is now not a way to look both up and down)
With parallel hierarchies, there can be time slices which overlap (as discussed in the call). There are a different ways to cope with it: 1) Set-up the equations to deal with overlaps. As discussed in the call and setting up the equations to function correctly when there are overlaps is possible, but creates significant drawbacks (readability, speed of constraint generation, duplicates of equations blowing up model size). 2) Keep the equations as is (assuming we find a solution for the first bullet), and assume the user is responsible for making sure that non-hierarchical time slices do not come together in the model. That is too much responsibility for the user, and will lead to errors in the model I believe. 3) Imposing checks to check whether the user has made errors in using his temporal structure. Also this might not be an easy task (might be difficult to guarantee that checks cover all situations + checks make code more complex.

The project is a constant struggle in finding a balance between being as flexible as possible and making things as easy as possible.

I believe that in this case, the drawbacks of enabling a non-strictly hierarchical temporal structure are bigger than the benefits. Every user of Spine model will read through (some of) the equations, and have to provide input data. Very little (if any) will have a strong desire to user parallel hierarchies I believe...

With respect to any model I know, the hierarchical temporal structure and what it allows us to do is already beyond state-of-the-art.

In GitLab by @Poncelet on Jan 25, 2019, 08:36

@Poncelet I thought the resolution could vary within a single temporal level? Say, we define the duration parameter of the temporal_level object as something like [1, 1, 1, ..., 1, 3, 3, ..., 3] (if you know what I mean). So the relationship between the node and the temporal_level should work for the above purpose?

Good remark Manuel. I realize this is an option, and I think it works perfectly fine if you work within one level. However, if you would have something like a UC model where in the first 12 hours, everything is done on an hourly resolution, and afterwards, you want to model your coal plants with a 3-hourly resolution, but your gas plants still with an hourly resolution. If we don't support jumping to a different level, this would mean that we would have to have two parallel levels with resolutions [1,1,1,1,...,1,1,1], and [1,1,1,..., 3,3,3] which feels a bit funny. Also, if for some reason, your resolution of one flow would decrease over time (1h->3h), and another would increase over time (3h->1h), it would be impossible to set up the temporal levels such that they are hierarchical (although the time slices themselves would still be strictly hierarchical). As a final note, if you use different resolutions and temporal levels, it could be "clean" or easy to name the temporal levels reflecting the resolution.

In GitLab by @Poncelet on Jan 25, 2019, 08:40

Exactly. The alternative is to have a class called time_step where the objects are individual time steps --that would be more logical from the EAV point of view, but totally cumbersome in my opinion.

Given that we currently do not know yet how we would get he flow_tuples and nodal_balance_time_slices reflected in the data structure in which time steps are no objects, I would like to better understand why we would not use time steps as objects. Like you said, it would be the most EAV way of doing it. And with this, it could all be relationships representing the temporal structure. The cumbersome part is about setting up the temporal structure in this case? Wouldn't it be an option to have a separate tool for that purpose?

In GitLab by @jkiviluo on Jan 25, 2019, 10:04

In the Spine Model side, there is no need for anything else (forgetting forecasts for now) than tuples that connect one level to another through time_step_pairing (Manuel is using time_step_tree in the wiki, but it's really just a pair). That's what the constraints need - there is no need to traverse the tree in Spine Model and any attempt to do so will lead to more difficult equations (I thought we agreed on this).

Given that, there is nothing that stops someone to do 1h-3h and 1h-4h as long as they meet only at the 1h level. This could be that electricity grid runs 1 hour resolution and a gas grid at 3 hour resolution while there is also an industrial plant that has 4h resolution data and it uses electricity as input (but not gas). There would be separate sets of time_step_pairing for both purposes created by Jump_All_Out. They are both still part of the same model and co-solved as they should be, since they connect through electricity.

Another example: we have a 1h resolution electricity grid and a 3h gas grid, but also a 24h gas production facility. If the gas production facility does not use electricity, it is sufficient to have 1h-3h pairings and 3h-24h pairings for those constraints where the respective flow variables interact. However, if the gas production facility happens to use electricity as input, there will be a need to have a 1h-24h pairing as well for the electricity flow from electricity grid to the gas production facility. In this way, the equations see only specific pairings between two levels and the equations don't need to worry about the tree.

There may be an issue on how to give the weights, since that might require hierarchy in some of Kris' examples. In any case, that would be on the Data Store side and would be translated into right weights for each pairing.

I don't see why we would stop anyone doing the above things. For now we can assume that anyone who embarks on that will know to avoid overlaps that don't work. Longer term, we can do a validation and an error for those cases where the overlaps are not valid - that won't be very hard. If there is still disagreement, maybe it's best to have a call.

In GitLab by @jkiviluo on Jan 25, 2019, 10:11

The cumbersome part is about setting up the temporal structure in this case? Wouldn't it be an option to have a separate tool for that purpose?

Yes, imagine that you have hourly data for 30 years (we do for hydro power) and you need to make several different time_step_pairings with that. A tool could certainly do this, but it needs to be built first. And even if there is a tool, I feel more comfortable that there is a reasonable manual way to enter the data.

Also, I haven't yet seen that those time_step objects would actually be useful in the Data Store side. They are needed (as sets) in Spine Model, but that's conveniently handled by JumpAllOut.

In GitLab by @Poncelet on Jan 25, 2019, 15:57

I think it might indeed be good to have a call.

As I need to think about the arguments anyhow, here already a bunch of thoughts:

Another example: we have a 1h resolution electricity grid and a 3h gas grid, but also a 24h gas production facility. If the gas production facility does not use electricity, it is sufficient to have 1h-3h pairings and 3h-24h pairings for those constraints where the respective flow variables interact. However, if the gas production facility happens to use electricity as input, there will be a need to have a 1h-24h pairing as well for the electricity flow from electricity grid to the gas production facility. In this way, the equations see only specific pairings between two levels and the equations don't need to worry about the tree.

1) This example is something which is supported by the strictly hierarchical structure. 2) I do not like this idea of having to make changes to the temporal structure of the model ad-hoc. In the hierarchical structure, through data you would specify that 3h is above 1h, and 24h is above 3h. From that, it can be derived that 24h is also above 1h. By having these relationships, a user does not need to worry about having to change the temporal structure (add a 1h-24h pairing in your example) when they change the model. Assume the user has gas-fired power plants (gas at 3h resolution, electricity at 1h resolution), and a 24h gas production facility. So in your proposal, the only required parings are (1,3) and (3,24). Now if the user decides to model his gas flows from the gas-fired plants add 24h resolution instead of 3h resolution -> he might forget to add the pairing (1,24) and model will give bizar results. So I wonder, what could ever be a motivation to not have pairings between time slices which are hierarchical? In the end, even in the case of parallel hierarchical trees, why wouldn't we pair all time slices corresponding to the same hierarchical tree?

In the Spine Model side, there is no need for anything else (forgetting forecasts for now) than tuples that connect one level to another through time_step_pairing (Manuel is using time_step_tree in the wiki, but it's really just a pair). That's what the constraints need - there is no need to traverse the tree in Spine Model and any attempt to do so will lead to more difficult equations (I thought we agreed on this).

This might be more on semantics. What I think is important is that:

if we have time slices which are hierarchical (e.g., 1h, 2h, 4h), that this set of tuples (however we decide to call it) contains all combinations, so (1h, 2h), (1h, 4h), (2h,4h) (even though the equations in a specific model might currently not use the (2h,4h) pairing, it needs to be there for when the user decides to make a change in the resolution of one of the flows. In that sense, I agree that it is a set of pairs. However, the set of pairs form a tree (or parallel trees). A set of tuples containing only (1h,2h) and (1h,4h) would mean that there is not a tree. So in that sense, whatever we call it, I think it should make a tree (or parallel trees). This set of tuples is typically used in the equations to find all time slices which coincide to some degree.
As agreed in the call, there is something more needed than a tuple that connects one level to another, and that is a convention to know that one tuple has a higher/lower resolution than the other. This information is used in the model, as certain constraints are generated for the timesteps corresponding to the involved flow having the lowest resolution, while other constraints are generated for the timesteps corresponding the involved flow having the highest resolution. I think we agree on this, but I state it here to avoid misunderstandings.

But what about the set time_slice_path, which goes both up and down? What should go there? Do we need two different time_slice_path sets (one for the 3h hierarchy and one for 4h hierarchy)?

I believe a possible solution to this would be to create just a single set. For instance, if we would have two parallel trees: tree A and tree B with:

A = (1,2,4,8,168)
B = (1,2,6,12,24)

The tuples in the set time_slice_path would then be all tuples which would be in the time_slice_path sets of tree A or tree B.

Example 1): constraint on instantaneous power output of a number of flows for common part of the parallel hierarchical trees (e.g., highest resolution involved flow is 2h). In this case, the models uses time_slice_path to find all flows having something in common with the 2h time slice (searches both tree A and B) -> OK.

Example 2): constraint on instantaneous power output of a number of flows for divergent part of the parallel hierarchical trees (e.g., highest resolution involved flow is 6h). In this case, the models uses time_slice_path to find all flows having something in common with the 6h time slice (searches only tree B) -> OK if the user is careful. If the involved flows would have 6h, 8h and 24h resolution => the constraint would only find the flows with the 6h and the 24h resolution, and hence the equation did not what was intended.

Example 3): Same as Example 2, but now the involved flow having the highest resolution is 12h. And assume the involved flows have resolutions of 12h,24h and 168h. Also in this case, the constraint would not find the flow with the 168h resolution and the equation will not do as intended, despite the fact that the considered flow resolutions in principle don’t overlap => IF we support parallel hierarchical trees, I think that we must somehow make sure that all defined time slices which could be hierarchical are linked to each other (in the example, it would mean that additional tuples would need to be added, namely (4,12);(4,24);(6;168);(12;168) and (24;168) (so not just combining all tuples in tree A and B. This adds some complexity to setting up the temporal structure.

Example 4): fix_ratio_out_in type of constraint which is generated for lowest resolution time slice involved. Similar to Example 3, if the lowest involved flow would be 12h -> need to adapt the set of tuples such that it also would find a 4h flow.

Example 5): similar to example 2). Assume the involved flows are 4h, 6h and 8h => the user makes a mistake leading to an error in the constraint (only 4h and 8h flows are found).

For all examples: without a strict hierarchy, it will also become slightly more difficult to determine the involved flow with the highest/lowest resolution (easiest way could be based on hierarchy, but difficult when two time slices would be in parallel trees).

For now we can assume that anyone who embarks on that will know to avoid overlaps that don't work. Longer term, we can do a validation and an error for those cases where the overlaps are not valid - that won't be very hard.

I disagree here, and this is for me the key reason why I prefer not to support parallel hierarchical trees. Understanding why overlaps do not work means understanding the full implications of the temporal structure on all equations. This is something we cannot expect from our users. And even if they are aware of it, making a small change somewhere (for a second forgetting the implications on the temporal structure) could lead to errors. Also, I don't think the validation is so straightforward, and if the user gets an error, then he still needs to understand where it comes from, and adapt his temporal structure or data.

Depending on the constraint, the flows which come together in a certain equation can be almost anything. Some constraints (e.g., fix_ratio_out_in) are on a unit level, and those are quite easy to see. For other constraints (involving unit_groups, commodity_groups) -> can easily have a lot of flow variables involved (of which the model would need to make sure that they all belong to the same hierarchical tree). -> 1) I think this is cumbersome, and 2) that means that if a user carefully sets up his entire model with two parallel hierarchical trees and suddenly wants to add an additional constraint in which some of the flows being in different hierarchical trees come together, the user cannot generate the desired constraint unless he changes the entire temporal structure of the model.

So if we support parallel hierarchical trees:

The user can (will make) mistakes: either we think it is his responsibility or we need to add checks and errors which the user should then try to understand (I don't like both)
Additional constraints a user did not anticipate before could lead to flows of divergent parts of two hierarchical trees coming together -> his model is broken
Specifying the temporal structure becomes more difficult (not simply merging tuples of tree A and B) - more difficult for us to implement, but even more importantly also more difficult for users to understand.

In GitLab by @Poncelet on Jan 25, 2019, 16:06

On the data representation aspect:

Yes, imagine that you have hourly data for 30 years (we do for hydro power) and you need to make several different time_step_pairings with that. A tool could certainly do this, but it needs to be built first. And even if there is a tool, I feel more comfortable that there is a reasonable manual way to enter the data.

What is the current manual way to do this? Copy a huge JSON string? Where does this string come from (setting this up manually also seems cumbersome)?

Also, I haven't yet seen that those time_step objects would actually be useful in the Data Store side. They are needed (as sets) in Spine Model, but that's conveniently handled by JumpAllOut.

About the convenient handling of Jump_all_out. Here is the main concern I have. Manuel did a very nice job. However, the concern I have is regarding the flow_tuples and nodal_balance_time_slice information for which we haven't seen how Jump_all_out can/will handle this. If it can be done -> great. In that sense, throwing time slices as objects back on the table was maybe a bit premature. However, if it turns out not to be so easy, I'm saying we might want to reconsider the EAV approach.

In GitLab by @manuelma on Jan 25, 2019, 16:19

What is the current manual way to do this? Copy a huge JSON string?

You can use a julia expression such as [1:10000...] to generate a list from 1 to 10000, you don't need to copy anything.

Given that we currently do not know yet how we would get he flow_tuples...

Do you need the flow tuples, or do you need just a set of time steps in the end? If you need the set of time steps then it's enough to create a relationship between the commodity, node, unit, direction and temporal level. Passing the first four to the relationship-access function would give you the temporal level, and then you can access the time steps with the time_step parameter.

If you want to use the pure EAV approach for the same purpose you would have to create 10000 rows in the object table, plus 5*10000 more rows in the relationship table. True, you can do it through a script, but anyways, is that something we want to carry around in our database?

In GitLab by @Poncelet on Jan 29, 2019, 15:03

Update after call with Juha:

We decided to support parallel hierarchical trees in Spine model under following conditions:

Hierarchical weights can still be supported (this is something commonly used in investment planning models and has a higher priority than supporting parallel hierarchical trees in itself).
A warning/error message is generated when users define the temporal structure with parallel hierarchical trees. The user can decide to ignore it, but is warned that he stepping into dangerous territory.
The user is responsible for making sure that the overlaps of time steps on different hierarchical trees do not come together in the model. If not too cumbersome, we might consider adding checks in Spine model in a later stage. E.g., if the user has a 3h input and a 4h output flow for a certain unit, and adds a fixed_ratio_out_in constraint between those flows -> the equations will not perform as intended, but it is the users responsibility to prevent it.

Furthermore, we discussed that:

we want to define sets of tuples for different constraint containing the tuples for which the constraint is to be generated (so do a precalculation). THis should improve readability of the code (equation), and efficiency of model generation (possible reuse of these sets of tuples)
We consider it an option to develop a user interface (tool) to generate the temporal structure information as presented in the database (e.g., make a tool to map a 'spatio-temporal block' representation to the representation of the temporal structure in the database as proposed by Manuel.

Proposed next steps:

Kris: check feasibility of incorporating the weights when using parallel hierarchical trees
Manuel: 1) develop proposal of data representation of flow_tuples and nodal_flow_balance_time_slices sets. 2) If weights turn out to be feasible in parallel hierarchical trees, see where the data representation as proposed needs changes.

In GitLab by @Poncelet on Jan 29, 2019, 15:14

Do you need the flow tuples, or do you need just a set of time steps in the end? If you need the set of time steps then it's enough to create a relationship between the commodity, node, unit, direction and temporal level. Passing the first four to the relationship-access function would give you the temporal level, and then you can access the time steps with the time_step parameter.

This could work if flows are not changing temporal level. (see earlier discussion https://gitlab.vtt.fi/spine/model/issues/17#note_5844 and previous comments). I really believe we should support jumping to different temporal levels within one optimization horizon.

If you want to use the pure EAV approach for the same purpose you would have to create 10000 rows in the object table, plus 5*10000 more rows in the relationship table. True, you can do it through a script, but anyways, is that something we want to carry around in our database?

So no, I would prefer not to carry this around in the database. However, if it turns out that we do not find a nice way to accommodate these jumps in temporal levels within the current proposal of the data representation, we need to think about alternatives (and EAV is one of them). But again, this might be premature. Let's first try to accommodate jumps in temporal level within the current proposal.

In GitLab by @manuelma on Jan 29, 2019, 15:43

Jumping between temporal levels sounds to me like having more than one 'temporal level' (or temporal block if I may) at the same 'level' of the hierarchy. These temporal blocks would be successive, but this isn't something we have described in the current version of the temporal structure, is it? It needs to be carefully accommodated.

I really like where this is going, but I'd really prefer to wait for the next revision of the temporal structure before continuing with my proposal for the data representation. I hope that's ok.

In GitLab by @Poncelet on Jan 30, 2019, 09:26

Some more on the implications of supporting parallel hierarchical trees.

If we enable parallel hierarchical trees, some implications and remaining issues are:

The user can make mistakes. We can warn the user when he has defined a temporal structure with parallel hierarchical trees that this might lead to errors. It would be the users' responsibility to make sure that the overlaps of time steps on different hierarchical trees do not come together in the model. Adding checks and clear error messages could possible be added later, but is not straightforward and could be computationally demanding and add lines of code users need to plow through.
We would need a different way to specify the temporal structure than in Manuels' current proposal). Up to Req.3 is fine, but the way the hierarchy would need to be specified (Req.4) would need to be different as in the current proposal. This might complicate things for us and the users
Entering/checking the parameters for the different time slices becomes more difficult. As discussed during a previous call, for modeling dynamics nicely in investment planning models, it is valuable (and more and more common) to distinguish between a) duration of a time slice, b) the number of times that time slice is repeated within its parent time slice - also called weight, and c) the total_duration of a time slice (combining duration, and number of times the time slice and the parent time slices are repeated). My idea here would be that the user enters the duration and the weight (number of times it is repeated within its parent time slice) for each time slice. The total_duration could then be calculated internally, making sure that it is consistent. This could be done as in following equation:

When we would have parallel hierarchical trees, and the set time_slice_tree(t,t') would contain all tuples of time slices where t' is somewhere below or at the same level as t, this calculation would be incorrect when parallel hierarchical trees are used (in some cases at least).

Example: Assume we have following time slices on different temporal levels [h1, h2, ...] ; [month1, month2, ...],[winter, spring, summer,fall];[5_year].
We would have two parallel trees: 5_year -> month -> hours, and 5_year -> season -> hour. The set time_slice_tree would contain (jan,h1) as well as (winter,h1). When calculating the total duration of h1, the equation would give duration(h1)weight(jan)weight(winter), so in this example 155 = 25 instead of 5.

One way to avoid this is would be that the user enters all parameter values of the time slices (so duration, weight as well as total_duration). But we would not be able to check it (unless we figure out a way to better handle these parallel trees (meaning that we somehow would need to generate multiple time_slice_tree sets -> time_slice_tree(A), time_slice_tree(B), ... I wouldn't know directly how to...

In GitLab by @jkiviluo on Jan 30, 2019, 10:17

@manuelma 'Temporal level' might be a confusing term, since as you point out they can be successive in time. It's really a definition of temporal structure that is applied to a certain period in time as well as locations. That's why I prefer 'temporal block' - and once the location is also defined it becomes 'tempo-spatial block' or 'temporospatial block' according to dictionary. We also used the word 'temporal stage' earlier, but maybe it would be consistent to use 'temporal block' and 'temporospatial block'.

There are two types of models - 1. everything optimized at once and these would typically contain only vertical 'temporal blocks'. And then there are 2. rolling operational models that could have higher resolution for the beginning of the model and lower resolution going forward (and even uneven aggregation of time periods - sometimes things are rather stable for a while and hence less time steps is sufficient until the system ramps again). These models would need a looping structure and would contain possibly both vertical and horizontal 'temporal blocks'. However, the first type of models is really a special case of the second where the loop is performed only once.

In GitLab by @Poncelet on Jan 30, 2019, 10:18

Less technical post on benefits and drawbacks of enabling parallel hierarchical trees.

What we offer to the user by enabling parallel hierarchical trees The ability to have overlapping time slices (e.g., a model with 1h, 3h and 4h resolutions instead of the strictly hierarchical 1h, 2h, 4h), under following conditions

overlaps of time slices (e.g., 3h and 4h) are not managed correctly in the model (i.e., if in any of the equations there is both a flow with a 3-hourly resolution and a flow with a 4-hourly resolution, the equation will most likely not be correct).
no immediate support to users to see whether they made a mistake somewhere leading to flows of overlapping time slices coming together in any of the equations. Note that preventing flows with overlapping time slices would come together is not so straightforward. Certain constraints (e.g., emission limit, GHG share) can contain many flows form many different units in many different regions).

Some other notes to frame these benefits for users by enabling parallel hierarchical trees:

It is important to note that the choice for a specific resolution should mainly be done for computational reasons and reduction of model size. Aggregation/disaggregation of data that is provided/available in a different resolution will be managed automatically. So even if data would be available in e.g. 3-hourly resolution, it could easily be changed to an hourly resolution, a 2-hourly resolutions or a 4-hourly resolution (with some approximations). Also, I'm not aware of data sources having these resolutions which could lead to overlaps. I'm not argueing that there aren't any, but the fact that I cannot come up with a good example of any does provide an indication as to how frequent this will occur.
The time_slice_group concept allows us to handle constraints which would need to be specified for groups of time slices. For instance, if we have a model with a strictly hierarchical temporal structure (e.g., (1h,2h,4h), we would still be capable of adding a 3-hourly constraint by creating time_slice_groups -> max_flow_out(u,c,tg=[1h1, 1h2,1h3]).

Given all the above, I strongly believe that there are and will be very few users (if any) for which the parallel hierarchical trees as we could support them will provide added value. And even if there would be a user for which it has added value, I believe the added value will be marginal (i.e., it would not be a significant drawback to conform to a strictly hierarchical structure).

Drawbacks to users by enabling parallel hierarchical trees

entering the data structure might become more complex (we don't yet know how to do with with parallel hierarchical trees. This is not supported by Manuel's current proposal)
model documentation becomes more complex
Cannot support calculation of total_duration based on weights and duration (or if entered by the user, would have no way to check whether data is entered consistently).

In contrast to the value of parallel hierarchical trees, I believe these drawbacks affect almost every user coming into contact with Spine model.

Impact of supporting parallel hierarchical trees

Need to design a new way to enter the temporal structure in the data
Visualization of the temporal structure would become more complex
Would need to find a way to make sure that all time slices which are hierarchical are managed as such. E.g., if a user defines two parallel hierarchical trees: (A -> 1h,2h,6h); (B-> 1h,3h) -> would need to make sure that it is clear that 6h is also hierarchical with respect to 3h. (unless we again also assume that all of this is the users' responsability)

I wanted to list this overview (particularly regarding the benefits) to make a final decision on whether or not we want to enable parallel hierarchical trees. In my opinion, the drawbacks are bigger than the benefits and my vote would be on not supporting hierarchical trees. @juha_k @DillonJ @manuelma @mihlema , might be good to have a final call on this?

In GitLab by @jkiviluo on Jan 30, 2019, 10:20

changed the description

In GitLab by @Poncelet on Jan 30, 2019, 10:32

Jumping between temporal levels sounds to me like having more than one 'temporal level' (or temporal block if I may) at the same 'level' of the hierarchy. These temporal blocks would be successive, but this isn't something we have described in the current version of the temporal structure, is it? It needs to be carefully accommodated.

It depends on your viewpoint. I don't see it like that. The temporal levels are fixed and each contain a number of time slices within them. In the current temporal description, flows are assigned to individual time slices via the set of tuples flow_tuples. These time slices can be on different temporal levels. That is why I argued that it is not sufficient to establish a relationship between (u,c,tlvl).

The term temporal block seems to indicate a subset of the time slices within one temporal level (so basically some kind of time_slice_group). In that sense, one could view the jump between time slices on different temporal levels also as jumps between temporal blocks on different temporal levels. As such, I think jumping between time slices is more generic/flexible than jumping between temporal blocks.

One more thing to think about for the data representation of the temporal structure: I believe the concepts we have of commodity_group, unit_group, node_group and time_slice_group is extremely flexible and useful. All the groups up to now have been objects. However, how would we see time_slice_groups working out if time_slices are not objects?

In GitLab by @Poncelet on Jan 30, 2019, 10:43

I have created a new issue #84 to discuss the data representation of the temporal structure.

I propose that in this issue, we only continue on the fundamental temporal structure used in Spine model (where I think the only open issue currently is whether or not to enable parallel hierarchical trees).

In GitLab by @manuelma on Jan 30, 2019, 12:29

There are two types of models - 1. everything optimized at once and these would typically contain only vertical 'temporal blocks'. And then there are 2. rolling operational models that could have higher resolution for the beginning of the model and lower resolution going forward (and even uneven aggregation of time periods - sometimes things are rather stable for a while and hence less time steps is sufficient until the system ramps again). These models would need a looping structure and would contain possibly both vertical and horizontal 'temporal blocks'. However, the first type of models is really a special case of the second where the loop is performed only once.

Thanks @juha_k for the clarification. I thought @Poncelet wanted to jump across 'successive' temporal blocks (or 'sequential', if I may) within the same optimization?

In GitLab by @Poncelet on Jan 30, 2019, 12:35

I thought @Poncelet wanted to jump across 'successive' temporal blocks (or 'sequential', if I may) within the same optimization?

If we have an operational model where we would want to have an hourly resolution at first and a 2-hourly resolution later, I think we should support doing it in two ways.

Having only a single temporal level and have varying durations attached to the different time slices in this temporal level
Having two temporal levels (hourly and 2-hourly) and being able to define the flow variables for time slices that are not all of the same level (e.g., h1, h2, ..., h12, 2h7, 2h8, ...2h24)

In GitLab by @jkiviluo on Jan 30, 2019, 13:39

I see the vertical hierarchy as something unnecessary complex. To me, this should be something more simple. One has nodes and units in the system that should be modeled at a specified time resolution (or time structure as the time steps may have varying length). The 'temporal_block' would define the time_steps, durations and weights for a particular block of time. JumpAll_Out or Spine Model will then interpret in pre-processing what are the needed tuples to restrict the constraints. The different temporal_blocks can be viewed nicely even in the current Graph View: example. Please note that the example is for units and nodes when in reality temporal blocks would be connected to nodeGroups and unitGroups when the system is larger.

If there are things that require more elaborate information (like a weight that is inherited between temporal levels), then let's define additional parameters on a relationship between those temporal_blocks. All the same, this too should be pre-calculated into correct tuples and weights in the pre-processing phase. I think this is the same functionality that would be in the 'req. 4' in the https://gitlab.vtt.fi/spine/data/wikis/Spine-specific-data-structure-(for-Spine-Model).

And if the user defines temporal resolutions that could create invalid constraints, then we give a stern warning. And later on we (someone) might even implement checks that would actually locate the invalid constraints.

The biggest benefit is that we have created a framework that is both simple to use and a non-restrictive. The possible draw-backs are that we might have to do bit more coding (once) and that the user can do things that don't work (but will get a warning in the dangerous territory).

In GitLab by @Poncelet on May 9, 2019, 13:39

Have hierarchy as optional (not yet implemented in SpineModel). Have agreement on temporal model structure

Open remaining issues/discussions at this point are:

how to determine through data when to invest
data representation and implementation of looping structure for rolling horizon type of optimization. See issue https://gitlab.vtt.fi/spine/data/issues/44

Closing this issue (for now).

In GitLab by @Poncelet on May 9, 2019, 13:39

closed

spine-tools / SpineOpt.jl

Temporal Structure Proposal #22