Support towards multiperiod and/or stochastic OPF

[frederikgeth]

A few colleagues and I are considering contributing to PowerModels.jl to support multiperiod and/or stochastic OPF. We streamed your JuMP developers meetup presentation.

You'd need (at least) one additional dimension for each variable in your (optimal) power flow model to represent time (multiperiod) or the decision stages & samples (multistage stochastic programming).

How would you attack this? Given the current implementation you'd need to add this dimension to each formulation in /form, or is there some JuMP syntax to avoid this?

Do you see unit-commitment type models (load, generation, storage) being part of PowerModels or separated into a separate project eventually? Just like the gas grid flow models are already?

Obviously such functionality would go far beyond the scope of the .m matpower case format, so how would you want to evolve PowerModels' internal datamodel and the transformations to/from .m and json? Or other dataformats?

[ccoffrin]

Hi @frederikgeth thanks for your interest in power models!

In terms of scope, for sure Stochastic OPF, Secure Constrained OPF, and UC OPF are in the scope of the PowerModels package.

My current idea to support problems like Secure Constrained OPF and UC OPF is to add the notion of network replicates, see https://github.com/lanl-ansi/PowerModels.jl/issues/44. As you suggest, all variable indexes would have have a new replicate dimension (which is 0 by default). Then, when one is writing a model, they can make multiple copies of the network and link the variables together with additional constraints. This is not the absolute best encoding of such problems, but I think it's a good balance of implementation flexibility and simplicity.

As for datasets, have you seen that PowerModels supports extensions to the Matpower format? Ideally we would be able to define some standardized extensions to the Matpower format so that they are cross-compatible with PowerModels and Matpower. Very large time series data can be prohibitive to this approach; but I would like to start by keeping the formulation and data management as simple as possible.

Another open challenge with these problem domains is curating a reasonable collection of test cases from the literature. I am coordinating this IEEE PES Task Force to provide a common location for such collections of test cases.

I'll also bring your inquiry to the attention of @yeesian and @kaarthiksundar, who have spoken to me about developing similar extensions.

[yeesian]

Kaarthik and I had indeed spoken about developing similar extensions, but don't yet have anything concrete. I'll just mention some of the things we've spoken about:

---

Without having to suffer the pain of collaborating and standardizing with others, you can already have your own custom solution: one idea (e.g. in https://github.com/lanl-ansi/PowerModels.jl/pull/109#issuecomment-306292969) is to encode the values like

%column_names% bus    mean    std  
mpc.uncertainty = [
    1         0   10.8
    2         0    9.7
    3         0   18.0
    4         0    7.4
    5         0    7.1
    6         0   13.6
    7         0   12.5
    8         0   17.1
    9         0   17.5
   10         0   19.5
   13         0   26.5
   14         0   19.4
   15         0   31.7
   16         0   10.0
   18         0   33.3
   19         0   18.1
   20         0   12.8
];

% correlation matrix
mpc.corr = [
     1     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     1     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     1     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     1     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     0     1     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     0     0     1     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     0     0     0     1     0     0
     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     1     0
     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     1
];

Which can then be processed by PowerModels.parse_file and PowerModels.build_ref, before you work with it in julia.

---

But we might prefer ways of representing

• the original/raw (time-stamped?) samples, from which the parameters (mean, variance, etc) were estimated, and/or
• more complex (e.g. graphical) models of dependencies between the uncertain variables.

One approach is to have (as fallback) a way of pointing to external resources (e.g. CRS in the GeoJSON spec) from within the network data dictionary, which can then have their own formats for representing the models of uncertainty.

---

But this is not a formal proposal/spec. We'll be happy to review and provide feedback on any proposals you might have.

cc: @lroald who has been working on such problems for quite a while, and might have some thoughts on what such extensions should look like.

[yeesian]

How would you attack this? Given the current implementation you'd need to add this dimension to each formulation in /form, or is there some JuMP syntax to avoid this?

Here is an example of how you might define your own model: https://github.com/kaarthiksundar/n-k/blob/master/stochastic/prob.jl#L1-L41. The general idea is to replace PowerModels.post_opf with your own function.

[frederikgeth]

Wow, lots of great ideas.

I hadn't seen that PowerModels supports extensions to the Matpower format. It seems like I can't access the data.html page? e.g. https://lanl-ansi.github.io/PowerModels.jl/stable/data.html#The-Network-Data-Dictionary-1

Being able to deal with time-stamped data would be a something I'd like; also, support for CIM.

[ccoffrin]

Just updated the docs. Try this one, https://lanl-ansi.github.io/PowerModels.jl/stable/network-data.html

[yeesian]

@kaarthiksundar and I just sat down to talk through this yesterday:

(i) Provide data formats and utilities only for working with parameterized distributions (e.g. multivariate normal), since those seem sufficiently unique to PowerModels. General code for parameter estimation from data samples belong better to a different package.

(ii) The data file for describing the uncertainty might be better off as a separate file, and not be bundled together with the original MATPOWER file, to decouple the network specification from the model of (forecast/error) uncertainty. Even so, it still is meant more as a description of the model of uncertainty (for software/planning-purposes) and not as an archival data format (c.f. point (1)).

(iii) The data format for describing the uncertainty will likely be JSON-based (rather than MATPOWER-based; based on https://github.com/lanl-ansi/PowerModels.jl/pull/53#issuecomment-264965424). But given that we haven't yet seen the GRG format (based on https://github.com/lanl-ansi/PowerModels.jl/pull/53#issuecomment-264976493), I don't want to be hasty in being overly specific here. It should also provide a natural means of being extended to describe the uncertainty over multiple time periods. One idea is to describe it as e.g.

[{"type": "MultivariateLaplace",
  "params": {"location": [...], "covariance": ...},
  "indices": [...],
  "source": "bus",
  "timeperiod": 1,
  "timeunits": 15}, // minutes
 {"type": "MultivariateNormal",
  "params": {"mean": [...], "covariance": ...},
  "indices": [...],
  "source": "pd", // or "qd", etc
  "timeperiod": 1,
  "timeunits": 15},
  ...
]

It has its drawbacks, e.g. in not being able to describe the correlations between time periods, and between different components/sources of uncertainty, but it might already go a long way (for a first iteration) in capturing many of the stochastic models that are out there in the literature.

(iv) PowerModels can then provide methods for building common stochastic models (based on whatever proposed format in (iii)) analogous to those for building up the deterministic models. We aren't yet sure if we should be combining the dictionaries into one single ref::Dict object, so we intend to keep them separate for now. The final user-facing workflow might look something like

using PowerModels
using Ipopt

network_data = PowerModels.parse_file("nesta_case3_lmbd.m")
stochastic_data = PowerModels.parse_json("nesta_case3_lmbd.json")

run_stochastic_opf(network_data, stochastic_data, ChanceConstraintOPFModel, IpoptSolver())

where ChanceConstraintOPFModel is some user-defined model.

[kaarthiksundar]

Adding to the previous comment, we intend to create similar json file formats for time stamped data like the demands at various buses. The format would go something like

[{"type": "load",
  "year": 2004,
  "month": 1,
  "day": 1, 
  "period": 1, 
   "pd": { {"bus_id": 2001, "value": 245}, {"bus_id": 2002, "value": 250}, ..... },
  }, 
  ....
]

We then want to add a parsing function like

load_data = PowerModels.parse_load_json(filename; start=MMDDYY, end=MMDDYY)

[ccoffrin]

Some initial thoughts. The primary motivation for PowerModels is to help researchers quickly and easily replicate established state-of-the-art results. It can also be used for R&D type of efforts, but those are probably best left to other packages that build on PowerModels (e.g. think JuMPChace, JuMPeR), rather than be part of the core package. That said, the core package should have a software design that makes building such extensions as easy as possible.

In the case of Stochastic OPF and Multi time period OPF, there is an important question of what are the established state-of-the-art formulations and their data requirements? Not being an expert in this area, it seems to me that a simple scenario-based two-stage formulation and Dan Bienstock's CC formulation are sufficiently established that they are ready for integration into PowerModels base. More complex extensions of these works are probably best left to PowerModels extension packages until a specific formulation is broadly accepted by the community.

Given that, I would focus on the simplest-possible data encoding that would allow one to replicate the results of these previous works. For example, @yeesian's original proposal for CC-OPF. For time series data, if possible, I would use matrixes of data that have no missing entries. Based on my non-expert view, the JSON formats proposed sound like rich extensible formats that would be good for pushing the R&D much further than the current state-of-the-art, which is outside the scope of the PowerModels core.

Regarding a Matlab data encoding vs a JSON data encoding. In Julia there are many advantages of going JSON over Matlab, but at least for the near future, any dataset that is not presented in Matlab matrix-like format will most likely be ignored by the broader power system community. That is a tradeoff that needs to be weighed carefully.

[frederikgeth]

I agree a simple scenario-based two-stage (preventive-corrective) formulation should be the focus of a stochastic (security-constrained) OPF reference implementation. But the framework should preferably be truly multi-stage from the start. Regarding data, typically, only the (un)availability of lines and generators are considered as random parameters in SCOPF, which reduces the amount of parameters needed to define scenarios. Also, you'd also have to define which subsets of controllable entities (load shedding, generator redispatch, phase shift position, transformer tap, capacitor bank switching, breakers ...) can participate in the preventive stage and in the corrective stage. Futhermore, given the scope of PowerModels, I'd find it reasonable that the scenarios are defined with their probability of occurrence (e.g. not generated by PowerModels based on failure rates).

A (deterministic) multi-period OPF is at the intersection of unit commitment / economic dispatch with OPF (active/reactive). Next to the cost calculations over the time horizon, the constraints spanning multiple time steps include

• rate limiter of changing phase shift positions / taps / switching
• storage system dynamics (dE/dT = P_in - P_out)
• ramp rates of generators

I like Kai Heussen's 'Power Nodes' approach for modeling storage and conversion processes (http://orbit.dtu.dk/fedora/objects/orbit:30653/datastreams/file_5759480/content) and it would be extendable to multi-energy network optimization.

Finally, the approach you took with the core network formulations (AC, DC, SOC, QC) could also be considered for the modeling of loads/generators/storage for specific variants formulations of economic dispatch / unit commitment / power nodes. But that implies we have to develop some definition of the boundary of grid modeling versus unit modeling. For instance the nodal power balance would then have to be defined as 'power flow through lines aggregated by node' = 'power flow generated by loads/generators/storage aggregated by node'.

[frederikgeth]

Regarding implementation of a two-stage SCOPF. You have the choice between tree-type (compact) and scenario-type multi-stage formulations. Tree-type have implicit and scenario-type have explicit nonanticipativity constraints (all variables in certain stage bound by equality across scenarios). See illustrations p.5. I quote from this document on p.8:

Note that the implicit memory-efficient model has essentially the same structure as of the explicit extensive form in equation (1). The explicit model has the potential to become very large for real-world applications since it includes (redundant) variables and constraints for all scenarios in every stage. The main purpose of the implicit formulation is to exploit the structure of the scenario tree to only generate variables and constraints for distinguishable scenarios at a given stage.

So this suggest that the scenario-type formulation would be a good starting point (still have to further study this article). For the scenario-type formulation basically all current variables are redefined with an additional scenario and stage (time) index, e.g. v[node] -> v[node, stage, scenario].

Problems can then be parameterized as:

• OPF: 1 scenario; 1 stage
• SCOPF: >1 scenario, >1 stage
• multi-period OPF: 1 scenario, >1 stage
• UC: 1 scenario, >1 stage, copper plate power model

StuctJump is suited for this application but supports only two-stage stochastic problem formulations. You can check out a SCOPF (AC) example. StructJump, w.r.t JuMP, offers additional interfaces to specific solvers which make use of the block structure.

Any comments?

[ccoffrin]

I agree that a two-stage SCOPF formulation is a good place to start with this.

One of the major challenges we will face with any SCOPF model is the details of the problem, especially in the recourse actions that are allowed or not. This competition, https://gocompetition.energy.gov/, has put a lot of work into the formulation, so I think we can draw inspiration from there.

In terms of implementation. My plan is to add a very general structure to PowerModels called replicates, which will allow one model to have multiple copies of the network variables. Specifically variables of the form v[id] would become v[id, replicate_id] where replicate_id is another identifier that the user to define. In the example you suggest, in the problem formulation you would make a collection of tuples of the form (scenario_id, stage_id), and them make replicates for each of these unique combinations. My hope is that this one general abstraction in can be used in a wide variety of ways as researchers explore many variants of UC and OPF problems.

[frederikgeth]

Ok, I think the network replicate_id is a good approach. It should suffice for exploring the main variants of UC and OPF - and should be easily built upon in PowerModelsAnnex.

The data input is indeed a challenge - you can have different sets of generators involved in the preventive stage than in the corrective stage. Furthermore, PST shifts are regularly considered as actions (decision variables) in a SCOPF context.

[ccoffrin]

Great, the proposal in #150 should easily support complexities, such has different generation sets, becouse each network replicate can be changed independently of the others.