zip loads integration

[jkrstulo]

Here's my first e-mail to Leon about zip loads integration:

I have just finished first version of zip loads integration in pandapower. I have tested it against GridCal on a GridCal's 5-bus case and I get the same results, you can have a look on a test ipynb. However there are some questions about integration of this new option through pandapower that I wanted to discuss with you (only constant current part is problematic, constant Z loads are seamlessly integrated through shunts):

• generally about integration to pf: I have looked through GridCal code and implemented it on the same way...so as we said, constant Z loads go as shunt elements to ppc['bus'] in columns GS and BS (no any problems, I've added them in _calc_shunts_and_add_on_ppc()). Constant current are elegantly considered by defining Ibus_cc vector, which is vector with constant current injections and it is needed in pf at two places:

  • dSbus_dV (and dSbus_dV_calc): where Ibus was Ibus = Ybus * V and now is Ibus = Ybus * V - Ibus_cc
  • _evaluate_Fx: where mis is now evaluated as: mis = V * conj(Ybus * V - Ibus_cc) - Sbus

• integration to ppc: the biggest question is now what is the most elegant way to get Ibus_cc vector in power flow calculation. In regard to inputs of _run_newton_raphson_pf(ppci, options), (no access to net) there are 2 (or 3) possible options :

  1. through ppci - I made it like this for now as it was the simple preliminary solution...although I am aware that you probably prefer not to touch structure of the ppc. So the idea is to add added 13th and 14th column to ppc["bus"] for real and imaginary part of the constant current vector
  2. through options - alternative to ppc changes could be writing it to options dictionary which seems to me more like a workaround
  3. through **kwargs (or additional input) so that if options["voltage_depend_loads"] is True

• integration to pf functions: in any case vector Ibus_cc will be defined in _powerflow or at the latest in _run_ac_pf_without_qlims_enforced, and afterwards it will go as one of the input parameters in functions: newtonpf, _evaluate_Fx, _create_J_with_numba, _create_J_without_numba, dSbus_dV and dSbus_dV_calc In order to avoid potential problems with calling any of these functions elsewhere, I've added it as a keyword argument Ibus_cc=None, and then in the functions where we use Ibus_cc I've defined it as: I = zeros(len(V)) if I is None else I so it does not impact calculations when options["voltage_depend_loads"] == False

• integration to net.load: in GridCal, input data for zip loads are defined as, power in MVA, impedance in ohm, and current in kA. I have looked at other software (e.g. Digsilent PowerFactory) and it seems to me maybe more logical and practical to set it with coefficients of each of constant power, impedance and current part. In that case we should add 3 new columns in net.load, lets say: const_p, const_z, const_i (please suggest if you prefer some more convenient names and way of defining). These values would be considered only when runpp is called as runpp(net, voltage_depend_loads=True) and in that case a test that const_p + const_z + const_i = 1 could be performed.

[jkrstulo]

Leon's answer:

About the ppc indexing issue: I do think its best to integrate in in the ppc, and we have in fact already started to use individual ppc columns for different reasons: unsymmetrical part of the impedance element as an own column, there are additional columns for the state estimation and now also for ZIP loads. I think it is now time to standardize this. Florian has already proposed to make a pandapower index file that defines the indices the same as in pypower.idx_bus plus our special pandapower columns, so that the bus/branch matrices are always the same size und no columns are used twice by accident.

So currently we would extend the datastructure as follows:

• 2 additional columns in bus for current load P and Q
• 1 additional column in branch for asymetric impedances
• Jan: how much non-pypower columns in which tables are you using for the state estimation?

and make an idx_branch and idx_bus file in pandapower with the new indices and always import from there instead of from pypower.idx_branch / pypower.idx_bus. There are also new matrices branch_sc and bus_sc for the short circuit calculation, which could also be integrated directly into the bus and branch matrix. However, we would probably want to only initialize those columns in case a short circuit is calculated, because there are 10 additional columns for bus and 9 for branch, and it would probably not be so nice to have so much unnecessary columns in each powerflow ppc. But I would maybe do this after the index-files are set up properly.

About how to integrate it in pandapower: I also like the solution that you showed now better than Z and I as input, since Z and I are very different on each voltage level and not very intuitive. What I don’t like about the solution however is that the user always has to pay attention that all three values sum up to one. So if I want to change the percentage of Z load, I have to change two values, which is a little awkward. I would therefore propose these two possibilities:

1. Variables p_kw, q_kvar, pz_kw, qz_kvar, pi_kw, qi_kvar -> the values for pz/qz and pi/qi would then be the nominal values at rated voltage. This is how it is already implemented for shunts and ward equivalents with pz_kw/qz_kvar, so it would be consistent
2. As it is now, only remove const_p and just fill in the value for const_p internally as 1 – const_z – const_i. This is easier to handle for the user but on the other hand there is no possibility to independently set values for p and q (is that something somebody would ever need?)

Both solutions would be fine for me, although I think I prefer solution 2. Comments from all sides welcome :)

As for the voltage_depend_loads parameter: do we even need one? Why would anyone define voltage dependend loads in the load table and then want to not consider them? Is there maybe a stability issue that the power flow could not converge with voltage dependencies and a user would like to disable it?

[lthurner]

Thank you Jakov! Little correction: I wrote 1 new column for asymetric impedance, but its actually 2 (real and imaginary part). Its currently hard coded as columns 17 and 18 here. Definitely not a pretty solution...

[jkrstulo]

integration to ppc I fully agree with you...adding additional columns to ppc is the best...and it only requires adjusting idx_bus file and naming the columns appropriately. about columns that correspond to constant-current loads: what we need is a complex value of a current, or it can be set with power (real and reactive) at the rated voltage. I don't think that is too important because user does not see that part? For now I will set 2 additional columns as real and reactive power at rated voltage for constant current loads. lets say named as: PDCC QDCC

integration to pandapower I am also more for option 2, mostly because of the following way of thinking that I follow: there is a load equivalent that consists of load mix...and than a user sets lets say 20% of it to const_z and and 5% to const_i. That seems to me more intuitive than independently setting kW or MW to each. About integration I again agree with you: 2 additional columns const_z and const_i initialized to 0, and const_p is internally set to 1-const_z-const_i (if you prefer they can be in %, so it columns can be called const_z_perc ?)

As for the voltage_depend_loads parameter: do we even need one? Why would anyone define voltage dependend loads in the load table and then want to not consider them?

it seems to me that voltage_depend_loads parameter could be useful for the following reason: image you have a large network with already set const_z and const_i for each load. And than, you want to check how much voltage-dependent part impacts your result in order to fine tune const_i and const_z, or, as you said, you have some convergence issues, than you can easily turn-off voltage dependence by this parameter. If you look at PowerFactory, there is also a checkbox to turn it on-off: so I would put it by default to True, open for a user to be able to set it to False.

[FlorianShepherd]

First: Thank you Jakov for the great work again. Leon also showed my the plotly integration. Wow 👍

About the idx files: I think it would be best to have some standard index file for the ppc, which contains the already defined indexes. For every submodule extensions to this could be imported. I imagine something like: from pandapower.std_idx import X from pandapower.submodule.extended_idx import Y

since we're discussing the standardization of the idx files and decided to integrate the pypower files (since the current PIP version is not compatible with numpy 1.12), It may be better to reorder the pandapower files in the main directory to increase readability. The current subfolders (converter, plotting, networks...) are extensions to pandapower, all remaining files are somehow classified as "framework". When I look at pp/tests, there is already some structure (api for example tests the toolbox, loadflow the powerflow and so on). Maybe we should try to keep that consistent. Here is a suggestion:

• converter, estimation, networks, plotting, shortcircuit, topology stay as they are
• powerflow will be added
• opf will be added
• core / framework / tbd will be added (contains build files, pd2ppc, idx files, results)
• pypower_extensions will be deleted and the files moved to powerflow / opf

I'm undecided about api in tests. Isn't an API a programming interface for other services to use the functionality of the software?

What do you say? Maybe I open another issue to discuss this?

@lthurner : I have to look into your suggestion about ZIP loads in detail

[lthurner]

about columns that correspond to constant-current loads: what we need is a complex value of a current, or it can be set with power (real and reactive) at the rated voltage. I don't think that is too important because user does not see that part? For now I will set 2 additional columns as real and reactive power at rated voltage for constant current loads. lets say named as: PDCC QDCC

As you say its just internal variable, so which way is most convenient for the implementation is fine with me

About integration I again agree with you: 2 additional columns const_z and const_i initialized to 0, and const_p is internally set to 1-const_z-const_i (if you prefer they can be in %, so it columns can be called const_z_perc ?)

Yes I like that idea!

it seems to me that voltage_depend_loads parameter could be useful for the following reason: image you have a large network with already set const_z and const_i for each load. And than, you want to check how much voltage-dependent part impacts your result in order to fine tune const_i and const_z, or, as you said, you have some convergence issues, than you can easily turn-off voltage dependence by this parameter. If you look at PowerFactory, there is also a checkbox to turn it on-off: so I would put it by default to True, open for a user to be able to set it to False.

Ok, makes sense

[jhmenke]

Jan: how much non-pypower columns in which tables are you using for the state estimation?

Estimation uses 6 additional columns in the bus ppc and 12 columns in the branch ppc. But since these calculations are not performed in parallel, there is no conflict between the columns added by individual calculation modules i think. It could only become a problem if the same built ppc was used for all calculations.

[lthurner]

Yes, we should differentiate between general network model columns and columns that are only specific for one submodule.

The shape of the ppc then has to be initialized with different number of columns depending on mode (pf, opf, sc, se), for example here for the bus matrix.

Asymetric impedance and zip load columns should be added in the general model, and special columns are added for short circuit and state estimation?

[jkrstulo]

about integration to ppc:

The shape of the ppc then has to be initialized with different number of columns depending on mode (pf, opf, sc, se),

yes, that seems great

Asymetric impedance and zip load columns should be added in the general model,

it makes sense...but than it means ppc would look different if a user choses voltage_depend_loads=True or voltage_depend_loads=False ? and than I need to include that parameter in to_ppc(net) also. I am still thinking about a way to bypass including some additional columns to ppc: but that make sense only if you are thinking of building some analysis over net and not over ppc: so that e.g. all the power-flow input parameters, like Ybus, Sbus, etc. are acquired from net directly... so that we have makeYbus(net), makeSbus(net), get_Ibus(net)... but that actually means eliminating ppc as internal format and completely detaching from pypower, what is not an option as I guess?

I will then include const_z_perc and const_i_perc, seems more transparent and suggests directly from the name that percentage is required, while only const_z leaves it unclear...

I have 1 more issue (or more a note) about representing results of PF with zip loads: I will modify calculation of net.res_bus.p_kw and net.res_bus.q_kw columns in order to consider zip loads, but within pypower's results I was not thinking to touch anything: voltages and power flows are ok, only I will need to change is generation at the reference node. so makeSbus will actually still return (Sgen - SconstantpowerLoad).

@FlorianShepherd thanks, I am glad you like plotly... Your indexing file seems great... about files reordering: what you mean makes sense...but I am not very competitive to talk about that...

[lthurner]

to keep it simple i would just always create those columns and just not use them when voltage_depend=False

[lthurner]

about res_bus.p_kw: i agree, it does not need to be changed in pypower, just in pandapower. You can look at shunt results for reference, it is the same there

[lthurner]

@FlorianShepherd the topics you are raising about restructuring the code deserve a seperate issue I think

[FlorianShepherd]

@lthurner I agree with that. I'd just like to have that settled before intregrating the idx structure and the integration of pypower. I'll open another issue... Edit: see #21

[jkrstulo]

There is still a question to clarify about zip load columns in ppc and option to use them or not in PF calculation:

Asymetric impedance and zip load columns should be added in the general model,

to keep it simple i would just always create those columns and just not use them when voltage_depend=False

At first it seems this is the best, but after looking more into it I realized it is not so straightforward: so ppc can be created according to const_z_percent and const_i_percent...afterwards if voltage_depend_loads==False, constant-current part can be added back to constant_power loads, but constant Z part is already added to GS and BS columns and cannot be easily reconstructed back (if there was a shunt on the same bus). Accordingly, I see 2 possible solutions:

1. use voltage_depend_loads as ppc option, so ppc will depend on that flag and than power flow will not even check value of voltage_depend_loads
2. use voltage_depend_loads as power flow option - but than we should add 2 more columns for constant Z part, which will than be added to GS and BS columns of ppc['bus'] if voltage_depend_loads==True. And if voltage_depend_loads==False than all the loads will be summed up to PD and QD columns. Thus, there will be in total 4 additional columns for zip loads in ppc['bus'] for active and reactive power for constant current and impedance respectively, named lets say as: PID, QID, PZD and QZD

Honestly, I don't see any option is much better, both have pros and cons...maybe first solution is slightly cleaner internally and requires more changes to pd2ppc, while the second one requires a pre-processing before running pf, which is maybe a bit awkward. what do you think ? please help :)

[lthurner]

I'm not 100% sure I understand the problem correctly, but maybe we could make internal power flow options const_z_percent and const_i_percent, and always set them to zero if voltage_depend_loads is False and to net.load.const_z_percent / net.load.const_i_percent if voltage_depend_loads is True. Would that solve the problem?

edit: i think this is basicly what you mean with option 1, i think this would be elegant and I don't see the downside to it? I would say that is definetly more robust than to have case-by-case analysis in the powerflow

[jkrstulo]

Yes, I am glad that we agree again...I was thinking a bit more about it and definitely option 1 is better. So different ppc will be generated depending on the voltage_depend_loads flag. Otherwise, in option 2, ppc would be always the same and than power flow would consider voltages depending on the flag, but this can be a bit cumbersome. Option 1 is preferred also knowing that ppc is internal format that an user should not interact with.

Ok, so I will implement it these days like that and than you can test it and see if it fits to all other functionalities.

[jkrstulo]

Hey people, I've finally solved all the issues that emerged in Travis after pushing all the work in regard to zip loads, so we can say a draft version is there. In any case there are some comments that I wanted you to know in order to be aware for the future development:

• in build_bus there are PCID and QCID defined as indices for 2 new columns in ppc["bus"] - they need to be added in the standard index file for the ppc (as @FlorianShepherd mentioned)
• I made a series of "manual" changes of number of columns and a new row types in _build_bus_ppc, _switch_branches, _branches_with_oos_buses: this could be eventually improved in the future with some more automatic definition of number of columns and new rows..supposing shape of matrices in ppc will be changing...
• zip loads for know work only for newton-raphson power flow and it is still not included in opf.
• in newtonpf dSbus_dV was imported from pypower. Since I've changed it in pypower_extensions, I included it here. So I believe importing dSbus_dV from pypower (this line) can be dropped out completely in the future?
• In brief there are know 2 additional parameters for a load which define constant impedance and constant current parts as a percentage of p_kw and q_kvar at rated voltage, that is: P = p_kw * scaling * (const_z_percent * V**2 + const_i_percent * V + (100 - const_z_percent - const_i_percent))/100

so this is more FYI, but any comments are more than welcome...

[jkrstulo]

one more comment:

• In rundcpp, runopp and rundcopp I've "manually" set voltage_depend_loads=False (there is no input keyword argument). For DC PF (rundcpp and rundcopp) that can be permanent solution since considering zip loads in DC PF does not make sense. Incorporation of zip loads in opf requires some additional work...

[lthurner]

Nice work Jakov!

Setting it to False in the DC functions is correct imho. In the OPF there are two stages: including ZIP loads as fixed loads and including ZIP loads as controllable elements. The first one should probably be easier as the second one, but still some additional work since it has to be represented correctly in the PF equations... I don't know the OPF implementation well enough to know how much work it is

[jkrstulo]

only to resume future TODOs for zip loads:

• integrate zip loads to other power flow algorithms (bfsw, Gauss-seidel and fdpf)
• integrate zip loads in OPF as:
  • fixed loads
  • controllable elements
• eventually for even further development phase: voltage dependent loads could be modeled with even more general exponential model, where instead of constant impedance (square voltage dependence) and constant current (linear voltage dependence) exponents for voltage dependance (e_aP, a_bP, e_cP in the following equation) would be user defined (current zip-model assumes e_aP=2, a_bP=1, e_cP=0):

[lthurner]

@ascheidl also added a check in runpp that disables voltage depencies if const_z_percent and const_i_percent are all zeros, so that the ZIP loads don't slow down the power flow unnnecessarily in case there are no voltage dependencies.

I think it is OK that ZIP loads only work with newton-raphson for now, but then we should raise a warning or even an error if another solver is used with ZIP loads, no?

[jkrstulo]

that's something I just started to do half an hour ago :) I am know just looking how to implement zip loads to bfsw. great that you did it already for the case const_z and const_i are zero...I will treat the non-NR algorithm case... in general how you prefer raising warnings using logging or raise warning?

[lthurner]

For logging warnings (i.e display a warning but continue to run the code) we use the logger module, for exceptions that stop the code we use raise , e.g. raise UserWarning("ZIP loads only work with NR")

[jkrstulo]

Ok, than I will use logger to warn that algorithm does not support zip loads and PF will continue with loads as constant power...

I see now that I've already included check to set voltage_depend_loads = False if const_z_percent==0 and const_i_percent==0, specifically here (or that was you :D )

[lthurner]

no, that was @ascheidl as mentioned above

[lthurner]

about the parameter name: abbreviating voltage_dependent_loads to voltage_depend_loads seems a little unnecessary, because its more unclear and only saves three characters. Can we name it either voltage_dependent_loads in full or maybe simply zip_loads?

[jkrstulo]

you are totally right...I didn't think about that...I agree with voltage_dependent_loads...I think it is more informative and universal than zip_loads which is more coloquial name...thanks

[lthurner]

I have added the new columns in the new pandapower index file here and imported them in build_bus and run_newton_raphson_pf. I didn't find it used anywhere else and it seems to work...

I think we can close this issue and maybe make a seperate issue for ZIP loads in the OPF?

[jkrstulo]

good work Leon! I've checked and it seems that is all where new columns PCID and QCID are used...

Yes, you can close this and open a separate issue for ZIP loads in OPF...I still didn't catch some tome to study that issue in more detail...as soon as I dive into it I'll let you know..