Import relationship parameters into JuMP #30

Closed spine-o-bot closed 3 years ago

In GitLab by @manuelma on Jun 27, 2018, 20:34

As of now we're able to import object parameters into convenience functions such as capacity which is called like this capacity(u) where u is the name of an object of the unit class.

But what happens if we define capacity as a relationship parameter for the unit_commodity class? This is related to issue #24. I would imagine that in that case we want the capacity function to be called like this:

capacity(u, c) where c is the name of a commodity object.

Note: I'm omitting here the 'optional' parameters t for time_step, s for sample and so on.

What do you think? Do you see a more convenient way of implementing this?

In GitLab by @manuelma on Jun 27, 2018, 20:35

changed the description

In GitLab by @manuelma on Jun 27, 2018, 20:36

changed the description

In GitLab by @manuelma on Jun 27, 2018, 20:36

changed the description

In GitLab by @manuelma on Jun 27, 2018, 20:36

changed the description

In GitLab by @Poncelet on Jul 2, 2018, 08:57

I'm not sure what would be the best way, but I do believe this is an important issue to figure out.

Also, whatever solution we come up with, it should be able to deal with parameters which can have 3 or 4 dimensions as well...

In GitLab by @manuelma on Jul 2, 2018, 15:05

@DillonJ do you have any thoughts on this? How should we access relationship parameters in JuMP equations?

In GitLab by @DillonJ on Jul 2, 2018, 16:33

In GAMS I used two dimensional sets to do this.

We have a parameter capacity which is defined on the relationship unit_output_commodity.

In GAMS I defined the set unit_output_commodity(U,C) which was a set of tuples that I could sum over and thus write:

Sum (unit_output_commodity(U,C), capacity(U,C))

So why not capacity(relationship_id, time_ref, sample_ref)... which makes it no different to referencing an object parameter, except we somehow have to get the relationship_id.

what if unit_output_commodity(unit_ref,commodity_ref) gave us the relationship ID to which we could then write,

capacity(unit_output_commodity(unit_ref,commodity_ref), time_ref, sample_ref)

But very often we will only have a reference to one object and we want a reference to the related objects also. I think you may have a way to do this already... but if not... maybe we can define something that allows us to write get_child_entity(relationship_class,parent_object_id) or get_parent_entity(relationship_class,child_object_id)...

or...

get get_child_entity(relationship_id) or get_parent_entity(relationship_id)

Just some rambling thoughts at this stage - I will need more time to think it through properly

Edit: I was responding based on the text in the email notification and just now noticed that you used the same example :-)

In GitLab by @DillonJ on Jul 2, 2018, 16:38

Let's not forget that we have also decided to support relationships between an object and a relationship, so take capacity(u, c), u (or c) could be a relationship - would that cause a problem?

This also doesn't get around the problem of where I get my reference to u and c. Do we have to iterate through every combination of u and c - that would be quite inefficient. That's why the sets of tuples in GAMS is nice because you can just loop through the pairs of entities that are actually related to each other. So this function to get related entities is quite important also

In GitLab by @manuelma on Jul 2, 2018, 16:51

I like it so far. We can implement two methods for unit_output_commodity:

the one we have so far that takes one argument and returns the list of related objects.
a new one, that takes two arguments and returns a reference to the relationship if any.

In GitLab by @Poncelet on Aug 7, 2018, 08:17

@manuelma @DillonJ : Do you have a view on the current status related to this issue?

We are currently implementing the second test system and this is one of the key things we want to advance in the second test system as we need multi-dimensional parameters for almost every conceivable model we want to build.

At the database side:

Is there a consensus on how such multi-dimensional parameters will be stored in the database? For example, let's say we have a parameter p^{MaxRatioOutputInputFlow}_{u,cg1,cg2}, where cg1 and cg2 are commodity group objects. Would that be a parameter of a relationship between an object (for instance cg2) and a relationship between u and cg1?

At the toolbox side:

Maybe not at its place within an issue for Spine Model, but is there already an idea as to how a user would enter such multi-dimensional parameter values via the toolbox? Directly following the approach above (i.e., first creating a relationship, then creating another relationship between the former relationship and an object, and finally adding a parameter for this relationship) is way to cumbersome. We need a user-friendly way to do this (and having this available would also strongly facilitate the model development/testing).

At the Julia side:

What are the functions currently available for accessing such multi-dimensional parameter values?

In GitLab by @manuelma on Aug 7, 2018, 09:17

@Poncelet I agree with what you say At the database side, for me that would be the most straightforward way.

Regarding At the toolbox side, I think we can let the user define a parameter by selecting one or more object or relationship classes that would be the 'dimensions'. In this case the toolbox would create the required relationship classes automatically. A similar interface could be created also for adding parameter values (please note the distinction I'm making between parameter and parameter value).

Now, At the Julia side, we can use the approach discussed earlier in this trend, but nothing is implemented yet. I can get into it during this week.

In GitLab by @Poncelet on Aug 7, 2018, 09:30

Thanks Manuel.

Quick follow-up:

At the toolbox side:

I like your idea of selecting the objects when defining a parameter. I do think that users should not have to select relationships here. I think the toolbox should (i) in some instances check whether the relationships are correct (e.g., if a parameter is entered with indices (plant1,electricity), it should be checked whether there indeed exists a relationship between plant1 and electricity), and (ii) create the necessary relationships where needed (at first sight, I think the relationships that need to be created are the relationships involving relationships).
There is currently no option available to create a relationship between a relationship and something else (I think this is the minimum functionality we need asap to get the data in the database via the Toolbox, and easily test the Julia equations, and the Julia functions to access these multi-dimensional parameter values.

At the Julia side: Great. Please coordinate with Maren and Steffen, who are working on the implementation of the second test system this week.

In GitLab by @manuelma on Aug 8, 2018, 08:38

It is actually possible to create a relationship between a relationship and an object using the toolbox. Check out the screenshots below.

In the first one I create a relationship class between unit and commodity called unit_commodity, and in the second I create another relationship class between unit_commodity and node called unit_commodity_node.

I can keep doing this forever, say create a relationship class between unit_commodity_node and another object class. We (or I) decided not to allow relationship classes between two relationship classes to keep things simpler for now.

So this method can be used for now to define multidimensional parameters, but I'll work on something less cumbersome.

add_relationship_class

add_meta_relationship

In GitLab by @steffenkaminski on Aug 9, 2018, 07:15

Nice feature!

Actually it was a little hard for me to find, because the relationship classes are not listed in the menu, which pops up, when you're doing: right click on object class -> add relationship class

@manuelma does it make sense to show the relationship classes in this menu, too?

In GitLab by @manuelma on Aug 9, 2018, 19:32

Yes @steffenkaminski I believe it would make sense to do that. We should create another issue in the toolbox project for that.

In GitLab by @manuelma on Aug 9, 2018, 19:35

I would like to hear @DillonJ 's opinion on how to define multidimensional parameters. We've been discussing about creating nested relationship classes using all the object classes involved in the parameter as 'dimensions', and then define the parameter against this relationship class. How do you like this approach?

In GitLab by @DillonJ on Aug 10, 2018, 12:23

@manuelma

I'm jumping into this after being away on vacation so I might be missing something. Here are some thoughts:

I think capacity(u, c) is ideal however the "trick" is in getting references to u and c? We could just loop through all u and c and do things only when capacity(u, c) has a value but I imagine that would be very inefficient. In GAMS we were able to define tuples, so we could, for example, loop over U_C which would contain only the Us and Cs that are related to each other. For example: total_capacity=sum(U_C(u,c), capacity(u,c))

So lets say we have the relationship unit_commodity, this tells us the Us and Cs that are related to each other.

You already have a mechanism to get the child/parent object in a relationship given one or the other. I can't remember the details - so let's say we define a function get_child_object(relationship(parent_object))

Then instead of capacity(u,c) we can write capacity(u,get_child_object(unit_capacity(u))

If I remember correctly, you had a neater way of doing this using JumpOUTs?

The other complicating factor is that we had said we would assume all parameter values are time varying, so we need a way to get these values as a function of the current time step and also, stochastic sample etc. where the parameter is stochastic.

In GitLab by @DillonJ on Aug 10, 2018, 12:35

Also, the above would need to bear in mind that a unit may be related to multiple commodities... we might need a way to somehow generate a set of related Us and Cs?

Not sure of syntax but something like:

[u,c] = get_relatives(unit_commodity)
totalcapacity=sum((u,c), capacity(u,c)

In GitLab by @manuelma on Aug 10, 2018, 12:37

Thanks @DillonJ for the reply. What you describe (if I understood correctly) is a mechanism for handling two dimensional parameters, and I think it's good, I just wonder how this can be extended to three-dimensional parameters and so on.

Maybe it's trivial, but I usually have trouble visualizing this sort of things.

In GitLab by @DillonJ on Aug 10, 2018, 12:40

Ah, yes, ok... first of all, I think we should avoid these to the maximum extent possible!

Is there an example we are working with to illustrate this? It is a bit of a nightmare!

In GitLab by @DillonJ on Aug 10, 2018, 13:08

I'm thinking that to fully evaluate the various options - we should try and write the full code block which would include how to iterate over the various commodities.

In GitLab by @manuelma on Aug 10, 2018, 13:09

Yes there is, see issue #32, the second test system.

In GitLab by @mihlema on Aug 10, 2018, 13:38

I am not sure whether I am addressing this to the right issue, but it seems related.

I just defined a relationship parameter with a certain value for transmission losses assigned to a relationship between a connection and a node. My database looks something like this:

However, looking at the JuMP_object function, it seems that there is no function yet to read this parameter from the database, is that correct?

In GitLab by @manuelma on Aug 10, 2018, 13:39

Correct @mihlema, needs to be implemented. I'm not sure how to do it yet but I'll give it priority now. Probably it will be based on the approach suggested but @DillonJ above, but other suggestions are welcome.

In GitLab by @Poncelet on Aug 13, 2018, 09:31

Some thoughts regarding the database/specific data structure for multi-dimensional parameters:

Ah, yes, ok... first of all, I think we should avoid these to the maximum extent possible!

I don't think we can/want to avoid this. When you want the model to be generic and handle multiple commodities, you quickly need parameters which have as indices unit, commodity1, commodity2, (,timestep, scenario).

We've been discussing about creating nested relationship classes using all the object classes involved in the parameter as 'dimensions', and then define the parameter against this relationship class. How do you like this approach?

Taking a step back and thinking about this, I'm not sure whether this use of nested relationships for the purpose of the multi-indexed parameters is the way to go. It feels like we are misusing relationships (i.e., they are not used anymore to indicate a relationship as such, but to be able to define certain parameters).

In all examples I can think of, it feels much more intuitive if the parameter is assigned to just one object (in most cases the node, or the connection). I understand that having a row in the database as follows is not sufficient, since the unit CCGT may have multple input/output flows and hence the value of 0.55 for the parameter MaxRatioOutputInputFLow cannot be interpreted by the model:

Entity | Attribute | Value

CCGT | MaxRatioOutputInputFlow | 0.55

However, I'm thinking there must a way to overcome this issue without having to resort to these nested relationships. Below just an idea for a possible alternative approach. To be honest, I'm not an expert in databases, and not fully aware of the current terminology/style guide, but I hope you get the idea.

Object classes:

Unit Commodity Parameter (new if not existing)

Relationship classes:

UnitInputCommodity UnitOutputCommodity UnitParameter (new if not existing) ParameterFirstCommodity (new if not existing) ParameterSecondCommodity (new if not existing)

Objects:

CCGT (Unit) Electricity (Commodity) Gas (Commodity) MaxRatioOutputInputFlow (Parameter)

Relationships:

CCGTGas (UnitInputCommodity) CCGTElectricity (UnitOutputCommodity) CCGTMaxRatioOutputInputFlow (UnitParameter) RatioOutputInputFlowElectricity (ParameterFirstCommodity) RatioOutputInputFlowGas (ParameterSecondCommodity)

Value assignment: RatioOutputInputFlow Value 0.55

As RatioOutputInputFlow is an object in the above example, it has a unique identifier, which can be used to find the relationships which make clear to which commodoties this parameter relates.

Happy to hear your thoughts on this kind of idea or alternatives.

In GitLab by @Poncelet on Aug 13, 2018, 09:31

In GAMS we were able to define tuples, so we could, for example, loop over U_C which would contain only the Us and Cs that are related to each other. For example: total_capacity=sum(U_C(u,c), capacity(u,c))

I agree with Jody that we need to have something like this (in the model description, I also frequently used such sets of tuples to write the equations). And this can easily be extended to multiple indices (each tuple in the set of tuples can have 2, 3, 4, ... elements). I'm wondering whether there is something preventing us from implementing this directly in Julia syntax? In the end, it should be possible to define a set of tuples in Julia I guess.

In GitLab by @DillonJ on Aug 14, 2018, 05:32

When defining our specific data model we should be aware that even if we can use relationship-relationships - this will make referencing the parameters Julia side (however we implemented it) more arkward and less complex. This is another reason we should strive to avoid using them when possible.

In GitLab by @jkiviluo on Aug 14, 2018, 06:13

Tuples are certainly necessary and sometimes they need to be multi-dimensional to facilitate clean code. I think the database users also need to be able to define parameters through multi-dimensional tuples (just like in GAMS). As an example defining reserves for a node would be like this:

p_nodeReserveParam(Leuven, Primary, gate_closure) = 6;
p_nodeReserveParam(Leuven, Primary, update_frequency) = 6;
p_nodeReserveParam(Leuven, Primary, reserve_length) = 24;
or even
p_nodeReserveParam(Leuven, Primary, upward, update_frequency) = 6;
...

Our generic data structure is not well equipped to handle this. Things would be simpler if we would allow multi-dimensional tuples in our parameter table. Probably limited to n dimensions. Something to consider.

In GitLab by @DillonJ on Aug 14, 2018, 06:36

@juha_k I think the fundamental generic data structure needs to be fixed at this point so we can progress the implementation.

In GitLab by @manuelma on Aug 14, 2018, 08:07

I like your idea @Poncelet but I wouldn't call an object class parameter. What about process? Then we can have the ratio_output_input_flow parameter defined against a process (and of course we'd have relationships process_unit, process_inputcommodity, and process_outputcommodity.)

Julia side we'd need to have something like this:

for p in process():
    u = process_unit(p)
    inc = process_inputcommodity(p)
    outc = process_outputcommodity(p)
    # do something with u, inc, and outc
end

With nested relationships we'd just have a relationship called unit_inputcommodity_outputcommodity, and Julia side something like:

for (u, inc, outc) in unit_inputcommodity_outputcommodity():
    # do something with u, inc, and outc
end

In GitLab by @DillonJ on Aug 14, 2018, 09:30

even better would be

for u,p in process_unit
    inc = process_inputcommodity(p)
    outc = process_outputcommodity(p)
    # do something with u, inc, and outc
end

In GitLab by @steffenkaminski on Aug 14, 2018, 09:59

@Poncelet , @manuelma I also agree with you. This approach seems to be much cleaner and easier to deal with than using nested relationships.

If I get it right setting one parameter would need following steps:

Creating parameter class in process class
Add process object, e.g. “obj1”
Create n relationships for this object using predefined relationship classes (process_unit, process_inputcommodity, process_outputcommodity, … )
Add parameter value to “obj1”

As it would costs a lot of effort creating/setting just one parameter a mask as shown in the figure below could be nice to have. Additionally, we could consider separating those multi-indexed parameters from the tree-view into another menu, as it could be highly confusing for the user and as there would be an object for each set parameter.

draft_menu_param

In GitLab by @pvper on Aug 14, 2018, 10:33

It might be hard to fit into the current database structure.

You would have to create a new object for every parameter, and since object names have to be unique you cant have the same name for the parameter, so the parameter name might have to be saved somewhere else
You would need a relationship class between the process and every other object class that has multidimensional parameters, as said above that would make the tree very unreadable in its current state (ex: unit_process, node_process etc.)
same for first_reference_object_class second_reference_object_class, you would need a lot of relationship classes for all possible input combinations for the parameter.
It's going to be harder to query for all values of a certain parameter for different objects, in the example above you would have to know which objects in the process class that corresponds to the RatioOutputInputFlow to be able to get all into the same table.

In GitLab by @manuelma on Aug 14, 2018, 10:56

sorry @juha_k what do you mean exactly by allowing multidimensional tuples in our parameter table? I imagine we can simply define ratio_output_input_flow against a unit object, and then put something like this in the 'json' field:

[
    {
        "inputcommodity": "gas",
        "outputcommodity": "electricity",
        "value": 0.8  
    },
    {
        "inputcommodity": "coal",
        "outputcommodity": "electricity",
        "value": 0.9  
    },
    ...
]

Is this what you mean more or less?

In GitLab by @manuelma on Aug 14, 2018, 13:47

So @pvper are you in favor of the nested relationship approach?

In GitLab by @pvper on Aug 15, 2018, 06:00

The idea would have to be refined, I think right now there are some issues that don't exist with the nested relationship approach. like order of indices, these are already in place for the nested relationship, with this new structure I think we would need to hard code the indices order of each parameter?

In GitLab by @jkiviluo on Aug 15, 2018, 06:45

@manuelma No, sorry for being unclear. I guess this should really go now to #36 - I'll repeat this there.

I thought we could add new object_id columns to the relationship table in the database. Consequently the relationship can have only n dimensions (n being the number of columns we decided to have as maximum - which is certainly not perfect solution, but I think it's the most simple).

In GitLab by @Poncelet on Apr 24, 2019, 07:28

closed

spine-tools / SpineOpt.jl