Closed spine-o-bot closed 3 years ago
In GitLab by @mihlema on May 6, 2020, 12:23
changed the description
In GitLab by @mihlema on May 6, 2020, 12:27
changed the description
In GitLab by @mihlema on May 6, 2020, 14:49
changed the description
In GitLab by @manuelma on May 6, 2020, 18:27
I guess we can also have different model
objects representing each rolling situation, and then find a way to alternate between them?
In GitLab by @Tasqu on May 7, 2020, 05:12
This would be my first attempt as well, since there might be other features the user might want to tweak between the day-ahead and intraday examples that have nothing to do with temporal_blocks
.
However, I suppose we would have to change a lot of the current implementation in any case, since we would need to have common variables between multiple models
that are currently stored in m.ext
. Depending on the desired interactions between the models
, defining these might be tricky.
In GitLab by @mihlema on Jun 11, 2020, 11:51
changed title from {-Complex rolling structures might not be possible with current temporal structure-} to {+Linking related optimizations+}
In GitLab by @mihlema on Jun 25, 2020, 10:57
I have been thinking about this a little bit also with respect to #187. I think I am in favor in having a concise framework on linking models that can also be extended later on by other models.
The first idea was to have multiple models within one db. But this would lead to some issues regarding our current model strucuture. Given that multiple models within one db would follow different temporal_strucutures, we'd need to link temporal_blocks to different models. That would in turn lead to the fact that either (1) nodes would be connected to multiple temporal_blocks or (2) nodes that appear in both models would need to be duplicated.
1 The first idea does not seem too great given the following example:
Unit_A
and Unit_B
are part of model_A
and model_B
, respectively. Both models do represent the same geographical "system" meaning that they both share node_both
. model_A
follows tblk_A
, model_B
follow tblk_B
. model_A
optimizes flows from Unit_A
to node_both
, and model_B
flows from Unit_B
to node_both
.
However, in this setting, there would be no way to tell the optimization of model_A
that unit_B
doesn't belong to it's optimization problem. Hence, we would need e.g. unit_model
relationships or a parameter on each unit to indicate which model it belongs to. - I find this not a great solution.
2 The second idea could work, but quickly become annoying. If we want to have a linking constraint between the models we'd need to enforce a mapping between the nodes e.g. node_both_mA
=> node_both_mB
. If both models share their complete geographical system, this seems very unintuitive
Another option would be link different models through different databases. But at the same time we would like to "have both models in spineopt at the same time" for performance (e.g. not writing back and forth to the db for the exchange between models and keeping the ability to use the model update step).
So far this brought me the following idea:
Model A, Model B, Model C share common data from source db (Note: I think this should rather be a "data pointer" rahter than an import, but I don't know how that would be done). Each Model db contains the information about the model structure and which variables are active for the respective model.
The config db (or maybe tool?) holds information about the sequence of models, iteration and termination criterion, linking constraints. For Benders decomposition this could look something like this:
Furthermore, for this example, we need to introduce in which sequence the models are executed, e.g. model_a first or something.
Next step would be to define the intersection between the models. model A -> model B: unit_invested; model B -> A: marginal costs. Of course we need to define when this information is transferred. Also in config we need to hold the information what the break criterion is - and when it will be evaluated.
I think, we can provide a standard config for popular things, in particular decomposition (as we're also promising this).
From a spine_opt perspective this generic approach would probably be a bit tricky and would require some significant changes to run_spineopt.jl
, as the information from the config needs to be translated into an execution script (I believe).
This idea is not quite finalized and there quite a few things that still need to be evaluated, for instance if we could even do this with run_spineopt.jl
but I thought it's good to think about a generic way of doing this. I believe this would leave the door open for a lot of flexibility for future models. Any thoughts @Tasqu @manuelma @DillonJ @jkiviluo ?
In GitLab by @Tasqu on Jun 25, 2020, 11:55
Personally, I'm not against using separate input DBs for different models
, albeit it sort of undermines the point of having a model
ObjectClass
to begin with. As you pointed out, the alternative would pretty much have to be adding the model
dimension to every single RelationshipClass
to avoid ambiguous definitions, which could get ugly. (Idealistically, it could be nice to be able to store the data and all the required definitions for a model
in the same database, but it could be too messy to be practical.)
I've previously given some thought to how the models run in a sequence could or should interact with each other, and to me the simplest way would seem to be having all the models
run using the same variable
dictionary. Since all the models
would use common variables
, there wouldn't be a need to externally define which variables
from model
A interact with which variables
from model
B etc., since this information would be "embedded" in the structures of models
A and B respectively. However, this approach would require very precise control over which variables
are saved from which models
, so that only the desired variables
from previous model
runs would be fixed for the next ones. Furthermore, the structures of models
A and B would have to be very precisely defined, and a more generic approach to constraining variables
using other variables
could be necessary, if we'd want to support constraining upper limits etc. in addition to simply fixing the variables
from previous runs.
An alternative would of course be to run some script between models
A and B to define how the results of A impact model
B. However, I'm not sure if this could be generalized in any manner, or if the script would have to be tailor-made on a case-by-case basis. Furthermore, we'd have to discuss whether these scripts would be parts of SpineOpt or not, since this type of general "task scheduling" starts to sound like Toolbox functionality.
In GitLab by @mihlema on Jun 30, 2020, 08:18
I think we can't assume that each model handles the same variables
. Even for Benders decomposition this is not the case (mp -> units_invested, m -> unit_flow
). We could have different run_spinemodels
on a case-by-case basis. However, this might become cumbersome for code maintenance. We have to carefully keep all run_spinemodels
update once we add functionality (e.g. new constraints, new variables).
In GitLab by @mihlema on Jun 30, 2020, 08:20
At the moment, run_spinemodel
does not support the linking of multiple models. Possible use cases are Benders decomposition for investment models, linking of sequential markets with different rolling structures, agent based models and so forth. The underlying algorithms can be different in their strucutre and have different "do's" and "while" conditions:
As it seems to be fairly difficult to incorporate multiple algorithms in one run_spinemodel
and as it would probably also not be the easiest to extend this functionality, we’d need designated run_spinemodel
’s on a case by case basis. - Which might cause some trouble in keeping all of them "in sync".
These run_spinemodel
’s do have some information in common though:
The only thing being completely different is the underlying algorithm. All algorithms could be described through a sequence of calculation specifications (as illustrated above for Benders and DA-ID coupling). We provide an “algorithm generator”, that evaluates the sequence of calculation specifications into lines of code. This algorithm is then written into run_spinemodel
, resulting in the following structure:
Initialization of temporal and stochastic structure
Initialization of variables, constraints, objective
Problem dependent algorithm
Writing of results
As the temporal_structure (and stochastic_structures) might differ between models (Benders, DA-ID), it is not sufficient to have a singleton temporal_structure in run_spineopt. Hence, the generate_temporal_structure()
would need to have a model as an input and return model dependent temporals
I am not sure how straight forward this is
The variables would need to be created for the different models. As proposed above these would follow from the database a model spans
Our constraint and objective functions are already indexed by a model index, the only thing we need to do is to run these functions for all models
This would require a framework on how to enter different algorithms. My idea is to represent (any?) algorithm within a toolbox tool, which is then translated into an algorithm. For a tryout for benders decomposition this looked something like this, where different while
and do’s
are linked.
From this tool, the algorithm is extracted and translated into lines of code, which are added to run_spinemodel
or called as a function from run_spinemodel
.
Note, alternatively, it’d also be possible to simply write multiple algorithm scripts and call them in run_spineopt
.
In GitLab by @jkiviluo on Jun 30, 2020, 11:57
I think your alternative #1 is much more simple than e.g. having multiple databases. Defining which units are in which model is something that should be done anyway. I guess we're currently assuming that based on which nodes are in the model. However, you could have two alternative units and choosing which one to use in which model run needs to be established.
In GitLab by @manuelma on Jun 30, 2020, 20:56
I have to say this looks quite interesting. In principle, any program (or algorithm) can be represented using a tree-structure, so it's not impossible to develop that structure within our framework.
I think we need to start with a very simple program such as if condition A else B end
and see what classes, objects, relationships and parameters we'd need to specify something like that.
In GitLab by @jkiviluo on Jul 1, 2020, 11:16
While that looks really cool, I'm afraid how much work there will be to make it functional and versatile. Current graph does not have directionality and supporting conditionality in the interface will mean quite a bit of work. And then when someone wants to do something not supported, the only way to make it work is to upgrade the interface with even fancier logic.
How about something more simple (at least first):
Allow including a Julia code snippet to the beginning of SpineOpt. The purpose of this snippet is to give the instructions how to loop and iterate between sub-models. This code snippet could be a parameter in model
, but it would really be a multi-line JSON. That code snippet would establish the logic between different models (loops and conditionals) - it would be calling SpineOpt with the right set of parameters and even parallelizing when possible etc. By default, it would just run SpineOpt as now.
E.g. a SpineOpt_decomposed
would be presented by single model entity in Spine Toolbox, but once it's inside SpineOpt it is divided into multiple sub-models as instructed by the code snippet. Since it's code, it gives lots of freedom to do whatever needed (and unfortunately also things that should not be done).
Another choice would be to establish a standard code framework for loops/decomposition etc. that will be fed with model
parameters that instruct how to implement the logic. This could be second stage - first make the code snippet to work - then we learn what kind of structures are needed and maybe we can generalize. After it's generalized, we can also think how to present the logic in a graph like above.
In GitLab by @manuelma on Jul 1, 2020, 11:29
I agree it may require more resources than we actually have, unfortunately.
I like your idea @jkiviluo, but I think we should stay as away as possible from passing code in parameter values. Just because of code injection. If we're going to go with this, I think we should develop that data structure to represent code: just basic branching and looping, if
and while
, might be sufficient.
In GitLab by @jkiviluo on Jul 3, 2020, 14:16
I'm ok either way. I was thinking that the code snippet would be the first step - something we can do safely now that there are no outside users. We get it to work first and then we can iterate it toward a data structure that directs the logic.
In GitLab by @manuelma on Jul 6, 2020, 20:45
@mihlema and myself did some brainstorming about this on slack. Based on that, I wrote a proposal here.
I think the proposal works, but it's just a first attempt, so any comments are welcome.
In GitLab by @DillonJ on Jul 10, 2020, 10:12
So I had a go at implementing the decomposition structure using the approach described here : https://gitlab.vtt.fi/spine/model/-/wikis/Custom-programs-in-SpineOpt
I think it works - but I have some thoughts that maybe we need to think through :
For example, here is the psuedo code for the decomposition structure I would like to implement :
setup_model(mp)
setup_model(sp)
while optimize_model(mp)
fix_mp_variables(mp, sp)
optimize_model(sp)
while roll_forward(sp)
optimize_model(sp)
end
save_marginal_values(sp)
rewind_model(sp)
end
write_report(sp)
Would it be another option to simply restructure the existing code and have this high-level optimization control flow structure written directly in Julia in a self-contained file which contains the main optimisation flow (shouldn't look much more complicated than the pseudo code above) and any custom functions for that flow (e.g. save_marginal_values()
is specific to the decomposition structure and it would live in the custom optimsiation control flow file.
Then we provide some mechanism to choose between alternative optimisation flows.
In GitLab by @manuelma on Jul 10, 2020, 11:32
The proposal was never meant to address 2, and I agree we should focus on that first, but that's a different functionality. Once we figure that out, we can think whether or not we want to create programs through data and go back to that proposal.
So we need a way to make models coexist in the same db, while sharing the 'spatial' structure of nodes, units, etc, but having individual temporal structures, is that the requirement more or less?
In GitLab by @DillonJ on Jul 10, 2020, 12:57
Perhaps we need to think it through more - and work through some use cases.
For example, during our case studies call, I was thinking of @mihlema's ideas for a multi-timescale market model that replicates day-ahead (and beyond) down to intraday and balancing. My first thoughts were that the different market timescales only differ in terms of the forecast error, temporal resolution and, perhaps, what constraints are included. My initial thoughts were that our flexible temporal and stochastic structure should be enough to capture all these differences and I wondered if different models were really needed to implement this - because in reality, day-ahead, intraday and balancing all form part of a single continuum and our structure is well suited to capturing this. However, the one thing we can't do now is have specific constraints apply during specific temporal blocks, but this seems trivial to do.
But what do you think of the idea of expressing the optimization flow in a single julia file - wouldn't it be a bit simpler? What does the data side system add? I feel that inevitably there would be something you would need to do that you can't specify using data but you can easily in a Julia script? My idea is that you would structure the current run_spinemodel code to include a single file that contains something like
setup_model(mp)
setup_model(sp)
while optimize_model(mp)
fix_mp_variables(mp, sp)
optimize_model(sp)
while roll_forward(sp)
optimize_model(sp)
end
save_marginal_values(sp)
rewind_model(sp)
end
write_report(sp)
In GitLab by @DillonJ on Jul 10, 2020, 13:00
basically, alternative run_spinemodel codes that are somehow selectable?
In GitLab by @manuelma on Jul 10, 2020, 13:04
I think I am already sold on that we don't need the program generator data side, at least not right now. I'm fine coding in Julia what we need. But for instance we need to learn how to do roll_forward(this_model)
and rewind_model(that_other_model)
and for that we don't have a proposal yet. We need to find a way to link models to temporal structures. Or maybe it's enough to keep one temporal structure per system and replicate it for each model in our code?
In GitLab by @mihlema on Jul 10, 2020, 15:43
So we need a way to make models coexist in the same db, while sharing the 'spatial' structure of nodes, units, etc, but having individual temporal structures, is that the requirement more or less?
Depending on the level or generality this might not be enough. An energy system can also be represented by different agents, that all have their own sub-problems. Each agent has different units associated.
We should at least leave the door open for this functionality.
But what do you think of the idea of expressing the optimization flow in a single julia file - wouldn't it be a bit simpler? What does the data side system add? I feel that inevitably there would be something you would need to do that you can't specify using data but you can easily in a Julia script? My idea is that you would structure the current run_spinemodel code to include a single file that contains something like
setup_model(mp) setup_model(sp) while optimize_model(mp) fix_mp_variables(mp, sp) optimize_model(sp) while roll_forward(sp) optimize_model(sp) end save_marginal_values(sp) rewind_model(sp) end write_report(sp)
I have been thinking about this before. I guess the main motivation for create this model_generator, ~as propossed here and in the wiki, is that we'd have a strong link between the data model object and the actual optimization model. To me, this seems fairly clean.
I agree that simply writing different execution script might be easy. However, we'd need (I guess) model_method parameters such as:
is_benders_subproblem(model=model1) = true
is_benders_masterproblem(model=model2) = true
In the code, we need to process this information. I guess it'd need to do something like this:
for mod in model()
$(mod).ext = .. #somehow splice model name here
initialize_temporal_strucutre($(mod)) ...
end
if !isempty(indices(is_benders_subproblem)) && !isempty(indices(is_benders_subproblem))
sp = filter(x -> is_benders_subproblem(model=m)==true, model()
mp = -"- #for master problem
setup_model($(mp))
....
For doing a similar thing for the market sequence we'd need:
is_day_ahead_problem(model=model1) = true
is_intra_day_problem(model=model2) = true
is_realtime_problem(model=model3 = true
And then the mapping as proposed above. Maintainability of this scripts can possibly become annoying.
I am already sold on that we don't need the program generator data side, at least not right now. I'm fine coding in Julia what we need. But for instance we need to learn how to do roll_forward(this_model) and rewind_model(that_other_model) and for that we don't have a proposal yet. We need to find a way to link models to temporal structures. Or maybe it's enough to keep one temporal structure per system and replicate it for each model in our code?
I guess the model sequence coming from the actual model object is the main advantage of the model generator proposal. roll_forward(this_model)
is literally given in the data.
In the end I guess the question boils down to whether we rather want new users to add new algorithms through script (1) of through the database (2).
(1) to do:
(2) to do:
I say new users, because I think that we will provide standard algorithms such as benders decomposition either way anyways. And for both ways, these standard example algorithms can serve as an example on how to do it.
In GitLab by @manuelma on Jul 10, 2020, 15:49
Those are good points, I generally agree. About this, roll_temporal_structure(this_model)
, the problem is currently roll_temporal_structure
takes no arguments, as the temporal structure is assumed to be unique and independent of any model. So we need to associate each model to a different temporal structure in Julia so they can be rolled and 'rewinded' independently. But how do we do that in data? Do we need model__temporal_block
, and how does it play along with node__temporal_block
for instance?
In GitLab by @DillonJ on Jul 10, 2020, 16:01
One point is that I'm not sure that there is a direct link between between the model object in the datastore and the MOI model object... right now, the model object_class really just holds high-level system parameters.
Let's say the model object class did correspond to the MOI model and the associated parameters were related to the MOI model... if we had multiple MOI models, it is likely they wouldn't have much commonality with respect to parameters... for example, if the master problem MOI model needs a "marginal value tolerance" parameter (or whatever) that is very model specific, but could be defined for every model which doesn't make sense.
That said, there may be parameters common to all models - e.g. convergence tolerance, max iterations etc... but we're not passing those anyway - we have to do that inside Julia right now.
So my feeling is that each model is rather an object class so you can define class specific parameters. We could treat objects within that class as alternative sets of parameters.
In GitLab by @DillonJ on Jul 10, 2020, 16:07
We also need to ask ourselves what the high level requirements are...
Do we need to support multiple rolling structures (we might, I'm not sure).
The decomposition structure in Example 2 assumes no rolling at the outer level - but if we wanted this - it woulnd't be just two rolling structures, it would be nested rolling structures- i.e. the model_window of the inner loop might get rolled... but then this is another layer of complexity and it's sounding onerous.
Then, would two side-by-side rolling structures be useful? In my decomposition structure, assuming no rolling at the outer level, all I actually need is an independent set of timeslices that covers the full model window (and no rolling).
In GitLab by @DillonJ on Jul 10, 2020, 16:14
Also - this question of agent based models... where now a model is actually part of your system... that is a really different kettle of fish - it's not really model linking in the sense that we understand it (but it is certainly linking models).
I'm not sure that the functionality that is needed for that is the same functionality that is required for the higher level linking functionality. But maybe it is - we need to brainstorm that. Different objects related to different models sounds messy. We could link a model to an object though, perhaps and have a different structure for that. and perhaps the additional data for that model lives elsewhere? A separate model DB perhaps?
Anyway - the point is we might need different structures for the higher level iterative linking and for the model within a model case.
In GitLab by @jkiviluo on Jul 13, 2020, 08:44
There are two different levels of model linking. One is the level of SpineOpt. There it should be feasible to setup a rule-based structure (model generator functionality). However, as Jody's example demonstrated, it might be wiser to start with code snippets that serve a specific modelling purpose and learn from those whether and how to generalize. I think this current issue should be just about linking within SpineOpt.
The other level is how to make different modelling frameworks to talk to each other. That needs to happen through Spine Toolbox. For instance, in EU project TradeRES that recently started, we need to make optimization models discuss with agent based models (and there is a multitude of different programming languages at play). Issue 757 is a proposal from last week how to handle that. It doesn't go into detail how to make models actually exchange information. Unidirectional links should be already feasible through current Toolbox workflow. Simple looping in Toolbox we've also discussed before (sub-DAGs) and that should enable most of the use cases. More complex stuff should be possible with scripting - that may be enough for now (or even during Spine project). But let's continue that discussion in that issue (https://gitlab.vtt.fi/spine/toolbox/-/issues/757).
In GitLab by @DillonJ on Jul 15, 2020, 08:42
So what is our next move here?
On our last call, we discussed rolling of the investment problem. Thinking about it a little further, in the context of the decomposed structure I don't believe this is necessary. This is because the master investment problem is actually relatively small compared to the operational sub problems since it is completely decoupled. And in fact, solving the master investment problem over a number of years all in one go, has many advantages such as optimizing seasonal storage better and timing investments better.
The master problem objective function is basically :
min sum( investment_variable(u,t1) * marginal_value(t2) )
for u in indices(candidate_units)
for t1 in investment_timeslices
for t2 in t_in_t(investment_timeslices, operational_timeslices)
)
So depending on your investment temporal resolution (which could be years or months), there aren't many variables at all in the master problem, so it is not necessary to roll it. Rolling of the investment problem is only really required for a conventional investment problem formulation that is simultaneously trying to optimise chronological operations (not the case in a decomposed structure, that's the point).
So in this case, it doesn't look like we have a use case for nested rolling?
If the below is the decomposition algorithm we want to implement, what is our next move given the current state of the model? The work I have done is now very far behind the current master and I would need to start from scratch.
Given that, for decomposition, a higher level rolling structure is not required, I found that all I needed to do for the below structure was create a second set of timeslices and related functions (e.g to_timeslice) that operate over the entire model window.
Thoughts @manuelma @mihlema @Tasqu
setup_model(mp)
setup_model(sp)
while optimize_model(mp)
fix_mp_variables(mp, sp)
while optimize_model(sp)
roll_forward(sp) || break
end
save_marginal_values(sp)
rewind_model(sp)
end
write_report(sp)
In GitLab by @DillonJ on Jul 15, 2020, 12:21
@mihlema a further thought... if we were to include agent-based models within the decomposed structure above - how would the algorithm look? I haven't worked with agent-based models and I'm not clear on the precise relationship between the agent models and, say, the operations problem - how often are they solved for example? Once a day? Once per-interval? Or, are they directly included in the operations sub problem - i.e. is it a single optimisation?
Perhaps you could write the psuedo code for an angent-based model so we know what would need to be supported if we want to at least leave the door open?
In GitLab by @manuelma on Jul 15, 2020, 13:17
Thanks @DillonJ that helps a lot. There's no apparent end condition for the outer loop though... How is that managed?
In GitLab by @DillonJ on Jul 15, 2020, 13:54
The idea was that optimize_model(mp)
would return false when some convergence criteria are met.
In GitLab by @manuelma on Jul 15, 2020, 14:04
I think we need model__temporal_block
anyways, right? So we can have static time slices for the master problem spanning the whole horizon, and rolling time slices for the subproblem.
So a first step could be to generate the temporal structure on a per-model basis and adapt all the related functions in SpineOpt so they get a model
object as argument. I propose I work on that now.
Once that's done, we can think of how to implement that particular pseudo-code above. Is the current version of run_spineopt
a particular case of it? Or they are completely separate? How does SpineOpt
knows it needs to run one or the other? How does SpineOpt knows which is the master model
object and which is the sub model
object, do we need an extra parameter for that?
In GitLab by @jkiviluo on Jul 17, 2020, 06:47
One type of agent-based model, that would be closest to SpineOpt, is one where the sub-problems are agents optimizing their individual profits and a master problem that represents the market clearance mechanism. In this case the problem would be solved as often as there are markets to clear.
I would think it would be easier to build a separate model for the purpose, but maybe borrowing constraints from SpineOpt. At least we shouldn't spend much time for an agent-based model in Spine project.
In GitLab by @mihlema on Nov 22, 2020, 16:45
changed the description
Issue can be found here: #318
The original issue
could not be created. This is a dummy issue, replacing the original one. It contains everything but the original issue description. In case the gitlab repository is still existing, visit the following link to show the original issue:
TODO