Open bergkvist opened 4 years ago
Thanks a lot for sharing your experience, much appreciated!
Could these two issues
net.line.c_nf_per_km being too high Too large values for net.line.r_ohm_per_km or net.line.x_ohm_per_km
Be tackled by just scaling down line length and looking at convergence?
This issue:
net.trafo.vkr_percent > net.trafo.vk_percent
should not affect convergence, since the real part is capped to the maximum internally. Its still a wrong modeling as you say, so it makes to flag it in the diagnostic.
The nan issue:
A single inf-value will make everything diverge. This could be handled similarly to c_nf_per_km (where an inf-value also will cause divergence)
could maybe be tackled easily by checking for nan in the admittance matrix? This would be an easy approach to flag that there is an nan somewhere - of course it would still be nice to pinpoint the exact elements with nans. This would also be really helpful for short-circuit calculations, where additional parameters are needed (e.g. xdss_pu for net.gen) that are often not available in systems that are made for power flow and lead to nans in the admittance matrix.
A single bad element can cause the entire powerflow to diverge. Some kind of binary-search (testing for convergence) with pp.select_subnet(net, buses=...) could be an interesting approach to locating bad elements.
This is great initiative, it would be very helpful to have something like that. We had a very similar idea some time ago, but as far as I know never implemented it. I think our idea was more to set all buses out of service except the ext_grid, and then setting buses in service starting at the ext_grid: first all buses that are directly connected to the ext_grid, then buses that are one branch away, two branches away etc. And then checking at which point the power flow fails, to find the culprit. Your approach looks very similar to that, although you are looking at the different voltage levels. But what about if there is only one voltage level? Or if you have narrowed it down to one voltage level, but that still consists of >100 buses? Of course with the approach described above, an open question would be how to tackle grid with multiple ext_grids... Do you have an idea how these approaches could be combined?
Could these two issues
net.line.c_nf_per_km being too high Too large values for net.line.r_ohm_per_km or net.line.x_ohm_per_km
Be tackled by just scaling down line length and looking at convergence?
This is a good question, but at least for r_ohm and x_ohm - falling below a certain threshold can cause divergence. See the plot below as an example of how scaling length_km both up and down can cause problems:
The white regions on the sides are nan-values due to divergence
It seems like it is generally safe to set c_nf_per_km to a value close to (or equal to) 0. I have yet to see this cause divergence. In fact, just setting c_nf_per_km=0 might be a very reliable check.
In the plot below you can see what happens to loading_percent just before it diverges as c_nf_per_km is scaled up. It changes very little/slowly for a while before just suddenly "blowing up":
Let's explore what happens if we scale r_ohm_per_km and x_ohm_per_km by different numbers. How does this affect convergence?
cig = nw.cigre_networks.create_cigre_network_hv()
def convergence_test(r_ohm_factor, x_ohm_factor):
net = cig.deepcopy()
net.line.r_ohm_per_km *= r_ohm_factor
net.line.x_ohm_per_km *= x_ohm_factor
try:
pp.runpp(net)
return 1
except Exception as e:
return np.nan
x = np.linspace(-12, 5, 10)
y = np.linspace(-3, 2, 10)
z = np.array([[ convergence_test(xi, yi) for xi in x ] for yi in y ])
fig, ax = plt.subplots(1, 1)
cp = ax.contourf(x, y, z)
ax.set_xlabel('line.r_ohm_per_km *= factor')
ax.set_ylabel('line.x_ohm_per_km *= factor')
ax.set_title('Cigre HV: Region of convergence')
plt.plot([0, 0], [y[0], y[-1]], '-k')
plt.plot([x[0], x[-1]], [0, 0], '-k')
Runtime: 16min 6s
Notice the split in the middle! If x_ohm_per_km is below a threshold for this network, it doesn't matter what value we set for r_ohm_per_km, as it will always diverge.
Runtime: 8min 10s
Notice that the region of convergence is a lot larger. In this first image we can't even see the "hole" around the origin.
And it turns out we need to zoom in quite a bit to even see it!
Runtime: 9min 25s
And to find the region of divergence around the origin, we now need to zoom around 25x further in compared to Cigre MV!
Turns out vkr_percent doesn't actually need to be larger than vk_percent to cause divergence!
Seems a bit more well behaved than the impedance-ROCs. If this is actually a perfect triangle, then maybe we could predict based on vkr_percent and vk_percent values whether the powerflow will diverge.
EDIT: Zooming out paints a slightly different picture. This was just the tip of an iceberg (or three):
Yeah, I think it would be helpful if the exact elements with nan-values could be pinpointed. This shouldn't be too hard to implement either.
def convergence_test(vn_kv_factor, c_nf_factor):
net = cig.deepcopy()
net.bus.vn_kv *= vn_kv_factor
net.trafo.vn_hv_kv *= vn_kv_factor
net.trafo.vn_lv_kv *= vn_kv_factor
net.shunt.vn_kv *= vn_kv_factor
net.line.c_nf_per_km *= c_nf_factor
try:
pp.runpp(net)
return 1
except Exception as e:
return np.nan
These regions of convergence look surprisingly weird/interesting.
To find the border of the "triangle"-tip of the iceberg with more precision, we can use a binary search:
```py import pandapower as pp import pandapower.networks as nw import numpy as np import matplotlib.pyplot as plt cig = nw.cigre_networks.create_cigre_network_mv() def convergence_test(vk_percent, vkr_percent): net = cig.deepcopy() # Deepcopy is much faster than loading the network again net.trafo.vk_percent = vk_percent net.trafo.vkr_percent = vkr_percent try: pp.runpp(net) return 1 except Exception as e: return np.nan # Binary search to find the limit between convergence and divergence for vkr_percent def find_vkr_percent_limit(vk_percent, vkr_conv, vkr_div, tolerance=1e-3): if abs(vkr_div - vkr_conv) < tolerance: return vkr_conv vkr_test = 0.5 * (vkr_div + vkr_conv) if np.isnan(convergence_test(vk_percent, vkr_test)): return find_vkr_percent_limit(vk_percent, vkr_conv, vkr_test, tolerance) else: return find_vkr_percent_limit(vk_percent, vkr_test, vkr_div, tolerance) # Assuming vkr_percent=0 converges, vk_percent=40 diverges, the limits will be found: vk_percent = np.linspace(0, 50, 151) vkr_percent = np.vectorize(find_vkr_percent_limit)(vk_percent, vkr_conv=0.0, vkr_div=40.0) plt.plot(vk_percent, vkr_percent) ```
Notice that up to some value for vk_percent, 0 <= vkr_percent <= vk_percent will cause converge. But we do actually get divergence here when vkr_percent > vk_percent.
The trailing 0s in the end of the plot means that no solution between 0 and 40 was found.
Very nice analysis! Adding some of this to the diagnostic function would be really cool. I'm curious if anyone has anything insightful to say about negative real line impedance? Pretty sure it's not physically possible. Negative reactance would, I suppose, be possible if the buses are close enough to have capacitance between them.
@jurasofish Thanks! The 2D-ROC-analysis is quite time consuming - so it might not be feasible to use on large grids. And you'd also need to know "where to look" (which I did using trial and error before finding good bounding-boxes for the ROCs).
For large grids, doing this on islands (or some other type of subgrid) could be an interesting approach.
Assuming you built a network whose model diverges. After all, it seems like "reality always converges" in some kind of way.
Your lightbulb might be rated for 40W - but that doesn't mean it always consumes 40W in practice. If the grid is not able to deliver enough power to satisfy your neighborhood - the bulb might glow less brightly, consuming less than 40W. "Reality adjusts your p_mw
-value down to make itself converge"
I suppose this specific example is already somehow dealt with in pandapower through voltage-dependent loads. I guess once the voltage drops towards 0 in some region, and is not even able to keep the lines/cabled electrified - then this will cause divergence in pandapower. Sort of like trying to model fluid flow in an empty pipe.
Essentially, as every individual parameter is scaled up or down - what is the intuition behind why the equations no longer converge, and how would "reality deal with it?"
parameter | if too high, will cause divergence because |
---|---|
net.load.p_mw | Network not able to deliver enough power from slack bus to loads |
net.line.c_nf_per_km | Network not able to charge up all the lines? My understanding of c_nf is that it corresponds to how much electric charge you have in a line. |
net.r_ohm_per_km | Restricts electron flow such that the network might not able to deliver enough power to the loads |
... |
parameter | if too low, will cause divergence because |
---|---|
net.line.r_ohm_per_km | Numerical instability due to how equations are solved. Would actually be fine in the real world. Can be fixed by using switch instead |
... |
I think creating some kind of overview like this as part of the documentation could be really useful.
I think rather than the slack bus or network not being able to supply enough power it might be more productive for you to conceptualise things in terms of voltage collapse. This will also help you understand why voltage dependant/constant impedance loads converge better. The slack bus (and the whole network) can always supply enough power - that's it's purpose.
You'd probably find a plots of average/min voltage in the network (pu) and slack bus power generation both versus line x/line r/constant impedance percentage/load size/etc. very informative.
Also do you mind me asking what your use case is for pandapower? Sounds like you're doing some interesting stuff.
@jurasofish I'm working for/writing my master thesis for Kongsberg Digital on this project: https://www.kongsberg.com/digital/solutions/kognitwingrid
Some of the networks I've been working on has more than 100,000 households/industries connected. (in the future, probably even bigger networks)
Data quality from the grid companies is not perfect - and so being able locate/fix problems in the model is important. The more automated the better.
One of the scenarios we are looking at is how the increased use of electrical cars in the following years will affect the power grid.
very nice, my experience is that LV network data is generally very low quality, as they were mostly installed before digital record keeping.
If you're looking at EVs you might want to also look at how their inverters can use reactive power to assist local voltage issues
Yeah, the LV data quality is indeed quite a bit worse than HV. Sometimes power transformers will be flipped the wrong way, lines/transformers are missing values. Voltage levels can sometimes also be wrong - and the grid might be partitioned/disconnected.
The data is generally exported in an XML-format (Common Information Model), where you have to follow a ton of references to get what you want. I've gotten pretty good at using pandas merge/concat as a result.
I don't know a lot about inverters (other than that they convert DC to AC, and are used in electrical cars/with solar panels). Do you know of any good articles/learning resources for what you are talking about?
What are you doing yourself related to pandapower?
It might be interesting to look at the absolute value of the impedance (radius), as well as its angle (theta) - instead of r_ohm and x_ohm directly.
r_ohm_factor = radius * np.cos(theta)
x_ohm_factor = radius * np.sin(theta)
As we noticed earlier, the factor will sometimes need to be very large, or extremely small to make a converging network diverge. Because of this, visualizing this on linear scales is inconvenient. A logarithmic scale might work better.
We can use a logarithmic binary search to find the upper and lower bounds for radius given a value of theta. In the plots below, only positive values for r_ohm and x_ohm are considered (0 <= theta <= pi/2
)
```py import numpy as np import pandapower as pp import pandapower.networks as nw import matplotlib.pyplot as plt def from_polar(r, th): return r * np.cos(th), r * np.sin(th) @np.vectorize def limit_polar_log_search(test_fn, theta, log_radius_conv, log_radius_div, tolerance=1e-3, max_iterations=50): # Check that radius_conv does not cause divergence if np.isnan(test_fn(*from_polar(np.exp(log_radius_conv), theta))): return np.nan # Check that radius_div does not cause convergence if test_fn(*from_polar(np.exp(log_radius_div), theta)) == 1: return np.nan while (np.abs(log_radius_div - log_radius_conv) > tolerance and max_iterations > 0): max_iterations -= 1 log_radius_test = (log_radius_div + log_radius_conv) / 2 if np.isnan(test_fn(*from_polar(np.exp(log_radius_test), theta))): log_radius_div = log_radius_test else: log_radius_conv = log_radius_test return np.exp(log_radius_conv) cig = nw.cigre_networks.create_cigre_network_hv() def convergence_test(r_ohm_factor, x_ohm_factor): net = cig.deepcopy() # Deepcopy is much faster than loading the network again net.line.r_ohm_per_km *= r_ohm_factor net.line.x_ohm_per_km *= x_ohm_factor try: pp.runpp(net) return 1 except Exception as e: return np.nan theta = np.linspace(0, np.pi/2, 300) radius_upper = limit_polar_log_search(convergence_test, theta, log_radius_conv=0, log_radius_div=50) radius_lower = limit_polar_log_search(convergence_test, theta, log_radius_conv=0, log_radius_div=-50) fig, ax = plt.subplots(1, 1) ax.plot(theta, radius_lower) ax.plot(theta, radius_upper) ax.set_xlabel('impedance scaling factor: theta') ax.set_ylabel('impedance scaling factor: radius') ax.set_title('Cigre HV: Upper and lower bounds for convergence') ax.set_yscale('log') ```
ext_grid(1pu) <--> bus(10kV) <--> line <--> bus(10kV) <--> load(1MW)
@jurasofish I've been trying to understand voltage collapse a bit more. As I increase c_nf_per_km, and look at the bus-voltages in a 2-bus network, this is what I see:
6000 powerflows/1min 47s
The blue line represents the bus connected to the slack bus, so it makes sense that the voltage here remains constant. Notice the "holes" in this line, however - showing where the powerflow diverges.
The orange line represents the load bus, at the other end of the line. The voltage starts rising, before diverging randomly and fluctuating between ~0.05 and 1.2-1.4.
Since Netwton-Rhapson can only converge to a single solution, maybe this means we actually have 2 solutions here, and it is unpredictable which one we will converge to? And then I guess this region might also be highly unlinear, meaning the solution can easily diverge by hitting a bump on the way to a solution.
Based on the 2D-ROC plots we have seen that the ROC sometimes seem to become fractal-like at the border (although in some places it is actually very smooth).
Does the situation with vm_pu = 0.05 in the result correspond to a "voltage collapse", like those that have caused several major power grid blackouts?
When there are multiple solutions, how would reality "pick one of them"? My guess is that both would be valid steady-states of the grid in reality. But in this region the network would be extremely sensitive to disturbances - that could throw it into a voltage-collapsed state.
Why does it seem like there isn't a continuous transition to the collapsed state?
I guess this might be because pp.runpp
is a steady-state analysis, and as a system is approaching a voltage-collapsed state, there is no steady-state "in between" that can be maintained over time. When a voltage collapse starts happening, it sort of becomes a chain reaction.
In the figure below, you can see that the system goes from having two possible solutions, to at some point having exactly one solution, before no solutions exist. The same simple system as in the previous post is used here.
Notice that the maximum power is achieved at vm_pu=0.5.
Divergence is caused by trying to use more power than the maximum power transfer theorem allows. https://en.wikipedia.org/wiki/Maximum_power_transfer_theorem
The two solutions simply correspond to the two possible load impedances that yield the same power consumption.
Some things to notice here:
These are problems that will typically always cause divergence (or have no sensible physical interpretation) if not fulfilled. An exception is the max_i_ka-rule - which will not cause divergence if broken, but cause nan-values for loading_percent on lines in the result.
from pandapower.auxiliary import pandapowerNet
import numpy as np
def assert_valid_network(net: pandapowerNet):
assert_valid_trafo(net)
assert_valid_trafo3w(net)
assert_valid_line(net)
assert_valid_load(net)
assert_valid_switch(net)
assert_valid_ext_grid(net)
def assert_valid_trafo(net: pandapowerNet):
assert (net.trafo.hv_bus).isin(net.bus.index).all()
assert (net.trafo.lv_bus).isin(net.bus.index).all()
assert (net.trafo.vkr_percent >= 0).all()
assert (net.trafo.vk_percent > 0).all()
assert (net.trafo.vk_percent >= net.trafo.vkr_percent).all()
assert (net.trafo.vn_hv_kv > 0).all()
assert (net.trafo.vn_lv_kv > 0).all()
assert (net.trafo.sn_mva > 0).all()
assert (net.trafo.pfe_kw >= 0).all()
assert (net.trafo.i0_percent >= 0).all()
def assert_valid_trafo3w(net: pandapowerNet):
assert (net.trafo3w.hv_bus).isin(net.bus.index).all()
assert (net.trafo3w.mv_bus).isin(net.bus.index).all()
assert (net.trafo3w.lv_bus).isin(net.bus.index).all()
assert (net.trafo3w.vn_hv_kv > 0).all()
assert (net.trafo3w.vn_mv_kv > 0).all()
assert (net.trafo3w.vn_lv_kv > 0).all()
assert (net.trafo3w.sn_hv_mva > 0).all()
assert (net.trafo3w.sn_mv_mva > 0).all()
assert (net.trafo3w.sn_lv_mva > 0).all()
assert (net.trafo3w.vk_hv_percent > 0).all()
assert (net.trafo3w.vk_mv_percent > 0).all()
assert (net.trafo3w.vk_lv_percent > 0).all()
assert (net.trafo3w.vkr_hv_percent >= 0).all()
assert (net.trafo3w.vkr_mv_percent >= 0).all()
assert (net.trafo3w.vkr_lv_percent >= 0).all()
assert (net.trafo3w.vk_hv_percent >= net.trafo3w.vkr_hv_percent).all()
assert (net.trafo3w.vk_mv_percent >= net.trafo3w.vkr_mv_percent).all()
assert (net.trafo3w.vk_lv_percent >= net.trafo3w.vkr_lv_percent).all()
assert (net.trafo3w.pfe_kw >= 0).all()
assert (net.trafo3w.i0_percent >= 0).all()
def assert_valid_line(net: pandapowerNet):
assert (net.line.from_bus).isin(net.bus.index).all()
assert (net.line.to_bus).isin(net.bus.index).all()
assert (net.line.max_i_ka > 0).all()
assert (net.line.length_km > 0).all()
z_per_km = np.sqrt(net.line.r_ohm_per_km**2 + net.line.x_ohm_per_km**2)
assert (z_per_km < np.inf).all()
assert (z_per_km > 0).all()
assert (net.line.c_nf_per_km < np.inf).all()
assert (net.line.c_nf_per_km >= 0).all()
def assert_valid_load(net: pandapowerNet):
assert (net.load.bus).isin(net.bus.index).all()
assert (net.load.const_z_percent + net.load.const_i_percent <= 100).all()
def assert_valid_switch(net: pandapowerNet):
assert (net.switch.bus).isin(net.bus.index).all()
assert (net.switch.element[net.switch.et == 'b']).isin(net.bus.index).all()
assert (net.switch.element[net.switch.et == 'l']).isin(net.line.index).all()
assert (net.switch.element[net.switch.et == 't']).isin(net.trafo.index).all()
assert (net.switch.element[net.switch.et == 't3']).isin(net.trafo3w.index).all()
def assert_valid_ext_grid(net: pandapowerNet):
assert len(net.ext_grid) > 0
assert (net.ext_grid.bus).isin(net.bus.index).all()
Lines/impedances constrain the maximum amount of power that can be transferred through based on reference voltage. Depending on where the ext_grid is placed, it might not be possible to deliver all the requested power. This causes the powerflow calculations to diverge.
Based on my current understanding - voltage collapse and the maximum power transfer theorem are closely related. To better understand if voltage collapse is the reason for divergence - a type of continuous power flow solution could be relevant.
def optimistic_network(net: pandapowerNet):
n = net.deepcopy()
n.line.c_nf_per_km = 0
n.trafo.pfe_kw = 0
n.trafo3w.pfe_kw = 0
n.load.p_mw = 0
n.load.q_mvar = 0
n.switch.closed = True
return n
pp.runpp(optimistic_network(net))
Hi @bergkvist ,
thank you for the detailed review of this issue.
When it comes to the maximum power transfer theorem, it doesn't seem very practical to implement in diagnostic. But it is interesting on its own. In diagnostic, the overload is covered by reducing the loads and checking the convergence.
The search for "islands" to identify unconverging sections of the grid can be very useful, especially in larger systems. The checks whether c_nf_per_km and vkr_percent are too high, as well as np.inf and np.nan, would be useful, too. The functions you are proposing also are looking great.
Can you please add those checks in pandapower diagnostic via a pull request? Also, please take a look at the overall structure in the implemented diagnostic module, so that the new checks fit into it.
Roman
Goal
Explore the debugging process of someone with a diverging network - trying to figure out how to make it converge and what they have done wrong.
Motivation
When I first started using pandapower around a year ago, I found it to be very hard to debug why a powerflow didn't converge. Especially after building a large grid. This is something I've gotten a lot better at now, so I want to share some of the tricks I've come across.
Problem
I find that pp.diagnostic often won't be able to figure out why a powerflow diverges.
From experience, there are several things that can cause a powerflow to diverge that the diagnostic tool is not checking for. Some of these things are:
net.line.c_nf_per_km
being too highThis is by far one of the things that seem to affect convergence the most. Trying to multiply this by 0.01/setting a threshold value could be a useful test.
net.trafo.vkr_percent > net.trafo.vk_percent
the real part of a complex number should never be larger than its absolute value. To fix this problem, I suggest validation logic in pp.create_transformer_from_parameters(...).
Since it is possible to change the value later, it should probably also be checked in the diagnostic tool.
The same is true for
net.trafo3w
, except here we have to check for 3 potential problems:net.trafo3w.vkr_hv_percent > net.trafo.vk_hv_percent
net.trafo3w.vkr_mv_percent > net.trafo.vk_mv_percent
net.trafo3w.vkr_lv_percent > net.trafo.vk_lv_percent
Too large values for
net.line.r_ohm_per_km
ornet.line.x_ohm_per_km
A single inf-value will make everything diverge. This could be handled similarly to
c_nf_per_km
(where an inf-value also will cause divergence)An approach to locating problem elements that pp.diagnostic is not able to pinpoint.
When you build a network with 100,000 loads, and the powerflow doesn't converge, it can be hard to figure out why.
A single bad element can cause the entire powerflow to diverge. Some kind of binary-search (testing for convergence) with pp.select_subnet(net, buses=...) could be an interesting approach to locating bad elements.
My personal approach
Define
net.bus.island
, as illustrated in the image below (partitions in the network could also be very relevant):Within an island, I ensure that every bus has the same reference voltage (vn_kv). Below you can see how I find the islands and iterate through different subnets to narrow down my search:
With the following helper functions:
To further narrow down the search within one of these subnets that I already know has diverged, I use shortest_path-subsubnets (I guess we need 2-subs here).