Investments - Next Steps #167

Closed spine-o-bot closed 3 years ago

In GitLab by @DillonJ on Apr 28, 2020, 10:07

During our discussion on investments, we broadly agreed that the next steps are as follows :

Link representative days tool with SpineToolbox/Model to generate requisite temporal blocks. The objective here is to run a representative period model that takes a Spine DB as input, does stuff, and outputs a temporal structure that implements the representative periods in a Spine Model.
Add investment variables to Spine Model.
This will involve determining the appropriate data structure and adding the investment variables to the Spine Model Code in a way that doesn't break existing models
Examine methodologies for dealing with discontinuities between representative periods suitable for modelling of long term storage and other long term constraints
Investigate a decomposition approach to include more operational detail

1 and 2 above should get us a basic implementation. 3. Will take us towards the state of the art and 4, in combination with the other aspects of the model (flexible temporal and stochastic structures) should represent a very significant capability.

Thoughts on this general approach? I will start working on 1 and 2 soon and will create specific issues for discussion.

@mihlema @jkiviluo @manuelma @junglegobslab @nnhjy @Tasqu @nhniina @CODWYER

In GitLab by @DillonJ on Apr 28, 2020, 10:08

changed the description

In GitLab by @DillonJ on Apr 28, 2020, 10:08

changed the description

In GitLab by @junglegobslab on Apr 28, 2020, 11:57

FYI I've written up my work so far on temporal representation and seasonal storage: temporal_representation_and_seasonal_storage.pdf. Hopefully it clarifies what I talked about during the last Skype meeting.

In GitLab by @DillonJ on May 11, 2020, 17:34

So I have the representative periods model working which is described here : https://gitlab.vtt.fi/spine/model/-/issues/175

I also have a version with investment variables working.

The basic idea is that the capacity variable is either a capacity (continuous variable) on integer, representing the number of units of a certain type. The below assumes you are doing one of the other and not both.

This is how I have gone about it so far:

New problem variables :

invested_units[unit] : the is the number of discrete units invested in by the model
invested_capacity[unit__to_node] : this is the amount of capacity invested in

Parameters :

candidate_units(unit) : this determines the maximum number of units that can be invested in. This is an upper bound on invested_units.
candidate_capacity(unit__to_node) : this determines the maximum amount of capacity that can be invested in
unit_investment_cost : the investment cost that is multiplied by invested_units in the objective function
incremental_investment_cost : the investment cost that is multiplied by invested_capacity in the objective function

Modified Constraints :

constraint_units_available: Units available is now number_of_units + invested_units

constraint_unit_flow_capacity:

              unit_flow < (unit_capacity + candidate_capacity)
                        * unit_conv_cap_to_flow
                        * (units_on + units_invested)

New Constraint :

constraint_unit_flow_capacity_invested:

              unit_flow < capacity_invested * unit_conv_cap_to_flow

Objective function :

add the following terms :

+ capacity_invested * incremental_investment_cost
+ units_invested * unit_investment_cost

Issues/thoughts :

Currently we assume a single (non-rolling) optimization window and the investment variables are not indexed by time
The above causes problems with much of the code that assumes a t-index. In particular, variable saving, and report writing.
For variable saving perhaps we either neither a window object or a timeslice that represents the full window - this would be useful for saving rolling window resolution results like objective costs etc.
Using a rolling window based timeslice could open the door to a two-level decomposed temporal structure
I have still to add connections into this but the ideas will be similar

The question is where we want to go from here...

Thoughts @mihlema @Tasqu @manuelma @jkiviluo @junglegobslab @nhniina

In GitLab by @mihlema on Jun 4, 2020, 08:36

                  unit_flow < (unit_capacity + candidate_capacity)
                            * unit_conv_cap_to_flow
                            * (units_on + units_invested)

If I read this constraint correctly, this would assume that invested units are always online. What if we do the following:

unit_flow < units_on * unit capacity #as is in SpineModel
units_on < units_available
units_available = units_historically_installed + units_invested #-units_decomissioned

Of course, that doesn't solve the problem for commodity flow specific investments.

In GitLab by @DillonJ on Jun 4, 2020, 08:42

Hi Maren,

Since our last telco, I'm going with a single investment variable: units_invested which can either be an integer or a continuous variable, in the same manner as units_on.

I think this is the most efficient approach

In GitLab by @DillonJ on Jun 4, 2020, 08:43

I created a new branch (jd_investments2) which is building off the latest dev with stochastics added, so it should be pretty easy to merge

Edit : so this is the branch you should look in for the most up to date code

In GitLab by @mihlema on Jun 4, 2020, 08:48

Ah, perfect,I see it on git. That's what I tried to sketch above. I think it's fine if we worry about the more advance investment later on. As I understand the current formulation doesn't allow to "upgrade" installed capacity for a unit but only enables the investment in a full unit - unless BOTH units_invested and units_on are continuous.

In GitLab by @DillonJ on Jun 4, 2020, 08:50

Yeah - I was thinking we get the basic framework into dev and then we all have time to get a feel for it and we can advance it to the next level then, and include mothballing, connections etc.

In GitLab by @mihlema on Jun 4, 2020, 09:00

Looking at

            + units_available[u, s, t] 
            == 
            + ( + number_of_units[(unit=u, t=t)] 
                + expr_sum( 
                    units_invested[u, s, t1]  
                    for (u, s, t1) in units_invested_indices(unit=u, stochastic_scenario=s,  t=t_in_t(t_short=t)); 
                    init=0 
                ) 
            ) 
            * unit_availability_factor[(unit=u, t=t)] # TODO: Stochastic parameters

I think we need something like (either as separate constraint or incorporated above):

units_invested[u,s,t1] = sum(
    units'_invested[u,s,t_inv,t1]  
        for t_inv in units_invested_indices(unit=u,scenario=s) if t_inv <= t1
    )
/forall u,s,t1

Maybe it'd also be good to rethink the name of units_invested. Maybe something like units_available_invested (not perfect either). And this one being expressed as:

units_available_invested[u,s,t1] = 
sum(
    units_invested[u,s,t_inv,t1]  
        for t_inv in units_invested_indices(unit=u,scenario=s) if t_inv <= t1
    )
- sum(
    units_EOL[u,s,t_inv,t1]   #EOL: end of life
        for t_inv in units_invested_indices(unit=u,scenario=s) if t_1 - t_inv >= lifetime[u]
    )
/forall u,s,t1

#NOTE: units_invested_indices would look back to the start of model run, taking into account investment decided on ipreviously

That being said @Tasqu , I'm wondering if/how this "looking back" to the year of investment would work with the stochastic structure.

In GitLab by @Tasqu on Jun 5, 2020, 05:31

Stochastics-wise, I don't think it should be an issue. Basically, the stochastics currently assume that t_history_t have the root stochastic_scenarios active on them, meaning the stochastic_scenarios without parents. As long as the constraint_indexes() function is written correctly to find the stochastic paths, accessing historical variables shouldn't become a problem.

However, I'm a bit concerned with extending t_history_t to cover really long periods if we have variables from potentially multiple roll_forwards ago that we have to access. The model could really start hogging a lot of memory if it can't drop any historical information from previous runs.

I'm also a bit worried with sums like this due to my GAMS background. Julia seems to be less sensitive to such things, but in GAMS a long sum over something like t_inv <= t1 would result in GAMS first going through all possible t to figure out which ones meet the conditional, and then going through the reduced set to form each constraint. In GAMS, this would quickly start taking a significant amount of time if there were a lot of ts in the model.

In GitLab by @DillonJ on Jun 5, 2020, 07:48

I think @Tasqu is right - with investments, the entire model horizon will be pretty close to the lifetime of a unit - so, for example, if you had a unit with a 25 year lifetime and a model horizon of 30 years, rolling one year at a time, you would conceivably have to sum over all previous rolling periods.

Instead, I think it would be better to have a parameter that keeps track of the current life of candidate units which gets updated on each roll_forward and incorporate this in the constraint instead.

Edit: and we might want to do this in a methodical way - an equivalent to preprocessing_data_structure.jl but which runs on each roll_forward. I can work on creating the bones of this is we think it's a good idea.

Thoughts @manuelma ?

In GitLab by @mihlema on Jun 5, 2020, 08:00

Thanks for the input Topi. I'm not sure though how explosive the memory consumption would be if it really is only calling the investment variable from saved previous rolls. Also we could only look as much back as necessary (e.g. max lifetime of units)

@Jody for me it's not clear how in your proposal you're keeping track of units that have already been installed in previous periods, if you are not looking back.

Edit: I think, if we want to incorporate de/mothballing this can potentially become even more complicated e.g. if the "lifetime-usage" is paused during the mothballing period, or the efficiency is impacted by it.

In GitLab by @DillonJ on Jun 5, 2020, 08:54

@mihlema Yes, we need to look back. Because the lifetime of units is comparable to the full modelling horizon in most cases, it is likely that we would need to look back over many previous rolls - and I don't think it makes sense to preserve the history of all model variables from many previous rolls just for the sake of investments.

So I am proposing to keep track of the lifetime usage of units in a way that doesn't require this - by creating a new parameter and updating its value on each roll.

In GitLab by @Tasqu on Jun 5, 2020, 11:38

Of course if we only saved the investment variable from way back, the memory wouldn't get clogged. However, if we handled this through t_history_t like any other time delays, all variable references would be included, I think.

It can be done, of course, but I'd prefer to avoid having different historical time steps specifically for investments, if it can be avoided.

In GitLab by @jkiviluo on Jun 5, 2020, 12:18

This would dispose the information how much capacity was invested at each time period. This can often be important for results.

Furthermore, if our investment variable spans e.g. 5 year blocks, it's going to be very little data that needs to be remembered (and calculated over in a possible sum).

In GitLab by @mihlema on Jun 5, 2020, 12:53

What if units_invest_indices() would just be equal to keys(m.ext[:variables][:invested_flow])? Than we would have current and previous investments covered and we can filter on this Dict for units that are end of life. Note: units_invest_indices() has (u, s, t_inv) as key

For a modelling timestep t: units_avail_invested_indices() = filter(x -> (t - x.t_inv) <= lifetime(x.unit), units_invested())

But this is also not 100% straightforward, as we still have to fix the values to the previous ones. Unless we make use of m.ext[:values][:invest_flow].... t.b.d. how this is then exactly implemented.

For an increase of performance we could also get rid of the entries that are end of life already - but I think we can worry about that in the next step. (e.g. have another Dict that incorporates all current variables and previous variables except for the ones that have already been identified as being EOL).

This could look like this:

units_invested_indices() = collect(Iterators.flatten(
    zip(collect(keys(m.ext[:values][:units_invested])),units_invested_now_indices())
    ))

where units_invested_now_indices() correspond to the variable investment and units_invested_indices() to all fixed and variables investments. Drawback with this implementation would be that in case of filtering, we have to filter an Array and if I call correctly @Tasqu that is not performant (I'm not sure how big of a deal this would be interms of investment variables, as they're typically of a low resolution).

In GitLab by @mihlema on Oct 20, 2020, 20:41

We are still missing the functionality to invest in connections and storages. Moreover, the unit_life_time constraint needs to be revised: At the moment the units_invested_available is always greater than the unit_lifetime. This seems a bit odd, given that we'd probably also need an economic lifetime, after which the unit is not useful anymore. Additionally we'd need to add mothballing to our formulation - which will probably be a bit tricky, as I belive, we need to refer back to the "used lifetime" before the mothballing. Lastly, I think we also need to take a closer look at the fixed operational costs. These are usually capital and land costs, and dependent on all available units.

Advances to the investment implementation are discussed in #211, #326, #325 #358 #359 Closign this issue

spine-tools / SpineOpt.jl

Investments - Next Steps #167