Passing stochastic paths to parameter calls, e.g. in constraint_connection_flow_capacity.jl

[mihlema]

I'm looking at the flow capacity constraint over here, in particular line 50 ff:

            - connection_capacity[(connection=conn, node=ng, direction=d, stochastic_scenario=s, analysis_time=t0, t=t)]
            * connection_availability_factor[(connection=conn, stochastic_scenario=s, analysis_time=t0, t=t)]
            * connection_conv_cap_to_flow[
                (connection=conn, node=ng, direction=d, stochastic_scenario=s, analysis_time=t0, t=t),
            ]

the s that gets passed to these parameters seems to be a stochastic path, rather than a stochastic scenario. I think we'd need to fix this in the future. The problem that I see here is that the summed connection_flows can have different stochastic scenarios; so which scenario should define e.g. the connection_capacity for the entire group (as the parameter can be stochastic too). Food for thought I guess. A similar issue is observed in constraint_total_cumulated_unit_flow_bounds.jl.

[mihlema]

Similarly, for the fix_ratio etc. constraints, a similar problem could occur, but here we avoid the problem by associating the parameter with the stochastic scenario of the right-hand side variable.

[mihlema]

I'll try to kick-off the conversation here. The tricky part is that we would need to decide, which scenario would take precedence in this situation. For the time slices, we user the lowest resolution as the defining time slice. But I don't think it's quite as straightforward for the stochastics.

We could have a convention that the constraint will be generated on the most "parent" of the involved scenarios as a start, and see if it suits our purposes. Any thoughts @Spine-project/spineopt-developers ?

[Tasqu]

Huh. I'm not even sure how SpineInterface.jl handles a parameter call with a stochastic path instead of a stochastic scenario. I'm guessing it just throws a key error or something? Any insights @manuelma ?

My intuition would be to try and tie all the parameters to the stochastic indexing of whichever variable they operate upon. E.g. here

- connection_capacity[(connection=conn, node=ng, direction=d, stochastic_scenario=s, analysis_time=t0, t=t)]
        * connection_availability_factor[(connection=conn, stochastic_scenario=s, analysis_time=t0, t=t)]
        * connection_conv_cap_to_flow[
            (connection=conn, node=ng, direction=d, stochastic_scenario=s, analysis_time=t0, t=t),
        ]
        * ((candidate_connections(connection=conn) != nothing) ?
           + expr_sum(
            connections_invested_available[conn, s, t1] for (conn, s, t1) in connections_invested_available_indices(
                m;
                connection=conn,
                stochastic_scenario=s,
                t=t_in_t(m; t_short=t),
            );
            init=0,
        ) : 1)
        * duration(t)

the parameters in question are attached to the connection_investments_available variable, if available.

I'm not sure how to handle parameters without a clear variable, though, as those situations could be interpreted in multiple ways. Currently, I believe we take the stochastic scenarios primarily from the variables, and then just find the applicable parameter data. E.g. If we have multiple scenarios for a variable but only one for a parameter, the same parameter value is used across all scenarios (assuming the stochastic structure and parent-child relations are set accordingly). Similarly, if the variable only uses one scenario while the parameter has a Map with data for multiple scenarios, only the data matching the stochastic scenario of the variable is used.

However, I suppose in some cases, one would like to generate the multiple constraints for a single variable using multiple scenario parameter data, in order to get something of a "worst-case" set of constraints. IMO, this is less useful, as the same thing could be achieved using a single pre-processed scenario, with aggregated scenario data from the parameter value Map, and thus avoiding generating multiple constraints where one would suffice.

[DillonJ]

@mihlema @Tasqu is this still an issue - has it been fixed?

[manuelma]

I'm not even sure how SpineInterface.jl handles a parameter call with a stochastic path instead of a stochastic scenario.

It cannot find the right key for the scenario in the map. It expects to receive a stochastic scenario, not an array of stochastic scenarios. It will fail.

We found one of these instances in #582 - it looks like there will be more of it along the way? @DillonJ

[clizbe]

@manuelma @DillonJ Has this been worked out or do we need to investigate further? I don't really understand how serious the issue is - just that something is tangled about stochastics.

[Tasqu]

@clizbe I took a quick look at the code, and as far as I can tell, this issue hasn't been addressed and the problem still persists. As far as I can tell, my above comment still applies:

Huh. I'm not even sure how SpineInterface.jl handles a parameter call with a stochastic path instead of a stochastic scenario. I'm guessing it just throws a key error or something? Any insights @manuelma ?

My intuition would be to try and tie all the parameters to the stochastic indexing of whichever variable they operate upon. E.g. here

- connection_capacity[(connection=conn, node=ng, direction=d, stochastic_scenario=s, analysis_time=t0, t=t)]
        * connection_availability_factor[(connection=conn, stochastic_scenario=s, analysis_time=t0, t=t)]
        * connection_conv_cap_to_flow[
            (connection=conn, node=ng, direction=d, stochastic_scenario=s, analysis_time=t0, t=t),
        ]
        * ((candidate_connections(connection=conn) != nothing) ?
           + expr_sum(
            connections_invested_available[conn, s, t1] for (conn, s, t1) in connections_invested_available_indices(
                m;
                connection=conn,
                stochastic_scenario=s,
                t=t_in_t(m; t_short=t),
            );
            init=0,
        ) : 1)
        * duration(t)

the parameters in question are attached to the connection_investments_available variable, if available.

I'm not sure how to handle parameters without a clear variable, though, as those situations could be interpreted in multiple ways. Currently, I believe we take the stochastic scenarios primarily from the variables, and then just find the applicable parameter data. E.g. If we have multiple scenarios for a variable but only one for a parameter, the same parameter value is used across all scenarios (assuming the stochastic structure and parent-child relations are set accordingly). Similarly, if the variable only uses one scenario while the parameter has a Map with data for multiple scenarios, only the data matching the stochastic scenario of the variable is used.

However, I suppose in some cases, one would like to generate the multiple constraints for a single variable using multiple scenario parameter data, in order to get something of a "worst-case" set of constraints. IMO, this is less useful, as the same thing could be achieved using a single pre-processed scenario, with aggregated scenario data from the parameter value Map, and thus avoiding generating multiple constraints where one would suffice.

TLDR: I could attempt to "fix" this by moving the parameter calls insde the expr_sum over connections_invested_available_indices used to resolve the stochastic_path into a stochastic_scenario, but I have no way of testing whether it would work as intended afterwards. Also, I'm not 100% that it even is the desired way to address this issue.