search

sogno-platform / dpsim

Real-time power system simulator including powerflow, (dynamic) phasors and EMT
https://sogno.energy/dpsim/
Mozilla Public License 2.0
67 stars 48 forks source link

Measure notebook runtimes #23

Open JTS22 opened 3 years ago

JTS22 commented 3 years ago

Some of the Jupyter-Notebooks using the new Python Interface (see #22 ) take multiple minutes for a full run-through, which causes the test pipeline to run quite long. This table aims to summarize the current run times (using the notebooks in #22 ) for evaluation and adaption of simulation run times and pipeline tests. The full notebook runtime is measured in a single run of pytest --durations=0 -v , the simulation run times are measured using the iPython %%timeit cell magic with the time calculated as the average over seven runs. Simulation run times contain the full setup and run of the dynamic simulation without PowerFlow or later data evaluation / plotting. Please note that this is not meant as a complete, accurate benchmark but rather as an overview over which notebooks should be tweaked when tested in the CI-pipeline.

Overall

61 notebooks in Circuits, Components or Grids
7 notebooks skipped in pytest(11,5%)
25 notebooks with asserts (41,0%)

Circuit-Examples

Full pytest-run in 760.53s (0:12:40)
26 passed, 2 skipped (92.9% tested)
10 of 28 notebooks with asserts (35,7%)

Notebook Pytest runtime Simulation runtime(s) Asserts Remarks
Compare_EMT_DP_Slack_PiLine_PQLoad 4.14s 1.73s (EMT), 419 ms (DP) DP vs EMT
Compare_EMTSynGenDQ7odTrapez DP_SynGenTrStab_SMIB_Fault Skipped 48.6s (EMT), 4.25s (DP) - Notebook contains no error, so test could be enabled
CS_R2CL print_attributes 2.79s 12.5 µs (EMT), 12.5 µs (DP) - Notebook exists to show off attribute print functions, so comparisons with SL got stripped
CS_R2CL 3.36s 11.8 µs (EMT), 11.4 µs (DP) EMT vs SL, DP vs SL
DP_Slack_PiLine_PQLoad_with_PF_Init 2.49s 71.6 µs - Plots currently plot the wrong time interval
DP_Slack_PiLine_VSI_Ramp 26.20s - - Notebook does not use dpsimpy, plot labels are in german
DP_Slack_PiLine_VSI_with_PF_Init 47.27s 45.8s - wrong time interval plotted, plot labels in german
DP_SP_SynGenTrStab_3Bus_Fault 32.01s 8.94s (DP), 8.52s (SP) -
DP_SP_SynGenTrStab_3Bus_SteadyState 12.71s 2.28s (DP), 2.11s (SP) -
DP_SP_SynGenTrStab_SMIB_Fault 17.57s 4.06s (DP), 4.11s (SP) -
DP_SP_SynGenTrStab_SMIB_SteadyState 12.29s 1.70s (DP), 1.45s (SP) -
EMT_DP_Slack_PiLineVSI Control_OnOff 34.25s 10.4s (EMT, control on), 11.2s (EMT, control off), 5.04s (DP, control on), 4.71s (DP, control off) -
EMT_DP_SP_Slack_PiLine_VSI 107.11s 11.1s (EMT), 48s (DP), 47.6s (SP) -
EMT_DP_VS_Init 3.12s 49.2ms (DP, set params), 88ms (EMT, set params), 45.1ms (DP, set attribute), 74.6ms (EMT, set attribute), 40.7ms (DP, set from node), 83.8ms (EMT, set from node) EMT vs DP DP vs EMT assert on set attribute commented out
EMT_DP_VS_RLC 19.35s 1.62s (EMT, VS+R), 518ms (DP, VS+R), 480ms (SP, VS+R), 2.82s (EMT, VS+RL), 785ms (DP, VS+RL), 856ms (SP, VS+RL), 2.79s (EMT, VS+RC), 758ms (DP, VS+RC), 865ms (SP, VS+RC) EMT vs DP vs SP
EMT_Slack_PiLine_PQLoad_with_PF_Init 2.77s 493ms - Plots the wrong time interval
EMT_Slack_PiLine_VSI_with_PF_Init_Params 69.97s 27.9s (Kp variation), 43.7s (Ki variation), 4.33s (final choice) -
EMT_Slack_PiLine_VSI_with_PF_Init 12.65s 11.3s -
EMTSynGenDQ7odTrapez OperationalParams_SMIB_Fault_JsonSyngenParams 93.01s 51.5s (no JSON-config), 50.4s (with JSON-config) JSON-config vs No JSON-config
EMTSynGenDQ7odTrapez OperationalParams_SMIB_Fault_ParamsModification 94.22s 50.8s (first parameter), 52.8s (second parameters) -
EMTSynGenDQ7odTrapez OperationalParams_SMIB_Fault 93.72s 52.9s (fundamental params), 54.5s (operational params) Fundamental vs operational params
EMT_SynGenDQ7odTrapez_SMIB_Fault 50.24s 53.6s -
RL_SourceStep_ParameterStudy Skipped - - Notebook does not use dpsimpy
SP_Slack_PiLine_VSI_with_PF_Init 3.69s 2.17s -
VS_CS_R4 3.39s 29.8ms (EMT), 52.4ms (DP) EMT vs SL, DP vs SL
VS_R2L3 3.47s 33.7ms (EMT), 63.1ms (DP) EMT vs SL, DP vs SL
VS_RC1 4.22s 19.9ms (EMT), 37.5ms (DP) EMT vs SL, DP vs SL
VS_RL1 3.77s 21.2ms (EMT), 33.9ms (DP) EMT vs SL, DP vs SL

Component examples

Full pytest-run in 133.57s (0:02:13) 10 passed, 1 skipped (92.9% tested)
7 of 11 notebooks with asserts (63,6%)

Notebook Pytest runtime Simulation runtime(s) Asserts Remarks
Diakoptics 2.91s 44ms (VS_CS_R4), 114ms (VS_CS_R4_Diakoptics), 56.3ms (VS_R2L3), 127ms (VS_R2L3_Diakoptics) -
Inverter_Grid_Test 49.46s 46s DP vs ML DP vs SL has no asserts, comparison is only plotted
Line 19.37s 904ms (SP Elements), 1.1s (SP Component), 951ms (DP Elements), 1.04s (DP Component), 2.13s (DP Diakoptics), 4.58s (EMT Elements), 6.14s (EMT Component), 79.9ms (Decoupling Line Elements), 109ms (Decoupling Line Component), 68.4ms (Decoupling Line EMT) Elements vs Component for DP, SP, EMT; DP vs EMT component
Slack 11.18s 433ms (SP Elements), 774ms (SP Component), 481ms (DP Elements), 850ms (DP Component), 938ms (EMT Elements), 1.27s (EMT Component) Elements vs Component for DP, SP, EMT; SP vs DP component, DP vs EMT component
SynGen_trStab_logger_test 1.40s - - This example exists to show errors in logger behaviour when multiple simulations interact with each other so timing it does not make sense here
SynGen_trStab 2.15s 8.7ms (SteadyState), 17.8ms (LoadStep) wr_gen(t=0) vs wr_gen(t=0.1)
SynGenDq7od_ParameterStudy Skipped 1.81s (DP ODE), 1.57s (EMT ODE), 1.34s (DP Trapez), 1.07s (EMT Trapez) - Tries to evaluate files which are not actually generated by the simulation
SynGenDq7od_SteadyState _DP_EMT 7.67s 1.81s (DP ODE), 1.44s (EMT ODE), 1.29s (DP Trapez), 1.07s (EMT Trapez) - SL results are loaded and compared to in plots, but there are no asserts
SynGenDq7od_ThreePhFault_DP_EMT 8.15s 1.84s (DP ODE), 1.55s (EMT ODE), 1.25s (DP Trapez), 1.09s (EMT Trapez) EMT vs SL
SynGenDq7od 6.77s 22.6ms (DP ODE SteadyState), 1.82s (DP ODE Fault), 1.31s (DP Trapez Fault) DP ODE Fault vs SL, DP Trapez Fault vs SL
Trafo 12.79s 736ms (SP Elements), 950ms (SP Component), 627ms (DP Elements), 851ms (DP Component), 2.03s (EMT Elements), 3.92s (EMT Components) Elements vs Component for DP, SP, EMT; SP vs DP component, DP vs EMT component

Grid examples

Full pytest-run in 223.14s (0:03:43)
18 passed, 4 skipped (81,8% tested)
8 of 22 notebooks with asserts (36,4%)

Notebook Pytest runtime Simulation runtime(s) Asserts Remarks
case9 1.80s 14.6ms -
case14 1.58s 26ms -
case145 6.60s 895ms -
case300 8.87s 2.0s -
CIGRE_MV_pf-interactive-dpsimpy Skipped - - There is no full simulation, only one step
CIGRE_MV_powerflow_profiles-dpsimpy Skipped 1.81s - Notebook contains no error, so test could be enabled
CIGRE_MV_powerflow-dpsimpy Skipped 24.4ms DPsim vs NEPLAN Notebook contains no error, so test could be enabled
CIGRE_MV_powerflow 2.54s - DPsim vs NEPLAN Non-dpsimpy version of CIGRE_MV_powerflow-dpsimpy, probably obsolete
cigre-mv-pf-profiles-shmem Skipped - - Does not run at all
DP_CIGRE_MV_withDG_withLoadStep 19.76s 14.3s -
DP_CIGRE_MV_withDG 17.84s 14.1s -
DP_CIGRE_MV_withoutDG 6.06s 2.13s -
EMT_CIGRE_MVwithDG withLoadStep 39.09s 36.0s -
EMT_CIGRE_MV_withDG 38.16s 35.0s -
EMT_CIGRE_MV_withoutDG 10.26s 3.92s -
IEEE_LV_powerflow 2.91s 539ms DPsim vs NEPLAN
PF_CIGRE_MV_withDG 2.52s 105ms -
SP_SMIB_SynGenTrStab_KundurExample1 19.56s - DPsim vs PSAT Does not use dpsimpy, instead calls CPP example as subprocess
SP_WSCC_9-bus_dyn_switch 12.09s 7.38s DPsim vs PSAT
WSCC_9-bus_dyn_switch 15.98s 9.22s DPsim vs SL
WSCC_9-bus_dyn 13.18s 9.95s Last value vs first value for all logged phasors
WSCC_9-bus 3.98s 998ms Last value vs first value for all logged phasors
rubensander commented 2 years ago

Grid examples

Notebook | Pytest runtime | Simulation runtime(s) | Asserts | Remarks -- | -- | -- | -- | -- DP_Slack_PiLine_VSI_Ramp | - | - | DP vs EMT ref | Notebook does not use dpsimpy EMT_DP_SP_Slack_PiLine_PQLoad_FrequencyRamp_CosineFM | - | - | DP vs EMT, DP vs SP | Notebook does not use dpsimpy
m-mirz commented 2 years ago

Next step is to remove unnecessary examples (especially C++) and reduce the runtime of the remaining ones.