Clustering isolated buses fails in simplify_network

[dhoop-iidev]

Checklist

• I am using the latest release
• I am running on an up-to-date environment

Describe the Bug

Hello everyone, I receive an "Assertion Error" during the simplify_network operation, more precisely when merging isolated nodes. It says that _In Bus cluster bus_id, the values of attribute busid do not agree and then lists some buses.

I want to create a model for Argentina. The only changes I made from the default config.yaml is changing the country to ["AR"], creating a new cutout (ar-2013), and change the snapshot to start: "2013-01-01" and end "2013-02-01". Thanks in advance for your help !

Error Message

INFO:snakemake.logging:
[Tue Oct 24 11:37:10 2023]
INFO:snakemake.logging:[Tue Oct 24 11:37:10 2023]
rule simplify_network:
    input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
    log: logs/simplify_network/elec_s.log
    jobid: 5
    benchmark: benchmarks/simplify_network/elec_s
    reason: Missing output files: networks/elec_s.nc, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/regions_onshore_elec_s.geojson; Input files updated by another job: resources/bus_regions/regions_offshore.geojson, resources/bus_regions/regions_onshore.geojson, networks/elec.nc
    wildcards: simpl=
    resources: tmpdir=C:\Users\le_re\AppData\Local\Temp, mem_mb=4000, mem_mib=3815
INFO:snakemake.logging:rule simplify_network:
    input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
    log: logs/simplify_network/elec_s.log
    jobid: 5
    benchmark: benchmarks/simplify_network/elec_s
    reason: Missing output files: networks/elec_s.nc, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/regions_onshore_elec_s.geojson; Input files updated by another job: resources/bus_regions/regions_offshore.geojson, resources/bus_regions/regions_onshore.geojson, networks/elec.nc
    wildcards: simpl=
    resources: tmpdir=C:\Users\le_re\AppData\Local\Temp, mem_mb=4000, mem_mib=3815

INFO:snakemake.logging:
This is the repository path:  C:\Users\le_re\Documents\PyPSA\pypsa-earth
Had to go 0 folder(s) up.
INFO:pypsa.io:Imported network elec.nc has buses, carriers, generators, lines, loads, storage_units, transformers
INFO:__main__:Mapping all network lines onto a single 380kV layer
INFO:__main__:Simplifying connected link components
INFO:__main__:Removing stubs
C:\Users\le_re\Documents\PyPSA\pypsa-earth\.snakemake\scripts\tmpypqh1c2l.simplify_network.py:306: DeprecatedWarning: aggregategenerators is deprecated. Use `aggregateoneport` instead.
  generators, generators_pnl = aggregategenerators(
INFO:__main__:Dropped 21 buses. A resulted load discrepancy is 4e+00% and 3e+00% for average load and generation capacity, respectively
Traceback (most recent call last):
  File "C:\Users\le_re\Documents\PyPSA\pypsa-earth\.snakemake\scripts\tmpypqh1c2l.simplify_network.py", line 1007, in <module>
    n, merged_nodes_map = merge_isolated_nodes(
  File "C:\Users\le_re\Documents\PyPSA\pypsa-earth\.snakemake\scripts\tmpypqh1c2l.simplify_network.py", line 833, in merge_isolated_nodes
    clustering = get_clustering_from_busmap(
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 534, in get_clustering_from_busmap
    buses = aggregatebuses(n, busmap, custom_strategies=bus_strategies)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 341, in aggregatebuses
    aggregated = n.buses.groupby(busmap).agg(strategies)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\groupby\generic.py", line 894, in aggregate
    result = op.agg()
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\apply.py", line 169, in agg
    return self.agg_dict_like()
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\apply.py", line 486, in agg_dict_like
    results = {
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\apply.py", line 487, in <dictcomp>
    key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\groupby\generic.py", line 297, in aggregate
    return self._python_agg_general(func, *args, **kwargs)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\groupby\groupby.py", line 1654, in _python_agg_general
    result = self.grouper.agg_series(obj, f)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\groupby\ops.py", line 1081, in agg_series
    result = self._aggregate_series_pure_python(obj, func)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\groupby\ops.py", line 1104, in _aggregate_series_pure_python
    res = func(group)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pandas\core\groupby\groupby.py", line 1640, in <lambda>
    f = lambda x: func(x, *args, **kwargs)
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 136, in consense
    assert (x == v).all() or x.isnull().all(), (
AssertionError: In Bus cluster bus_id, the values of attribute bus_id do not agree:
Bus
46      46
341    341
346    346
420    420
423    423
451    451
545    545
555    555
571    571
579    579
581    581
606    606
619    619
621    621
628    628
645    645
650    650
651    651
673    673
679    679
690    690
699    699
704    704
732    732
733    733
740    740
751    751
754    754
757    757
760    760
794    794
799    799
803    803
954    954
Name: bus_id, dtype: object
[Tue Oct 24 11:37:20 2023]
INFO:snakemake.logging:[Tue Oct 24 11:37:20 2023]
Error in rule simplify_network:
    jobid: 5
    input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
    log: logs/simplify_network/elec_s.log (check log file(s) for error details)
ERROR:snakemake.logging:Error in rule simplify_network:
    jobid: 5
    input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
    log: logs/simplify_network/elec_s.log (check log file(s) for error details)
RuleException:
CalledProcessError in file C:\Users\le_re\Documents\PyPSA\pypsa-earth\Snakefile, line 592:
Command 'C:/Users/le_re/mambaforge/envs/pypsa-earth/python.exe "C:\Users\le_re\Documents\PyPSA\pypsa-earth\.snakemake\scripts\tmpypqh1c2l.simplify_network.py"' returned non-zero exit status 1.
  File "C:\Users\le_re\Documents\PyPSA\pypsa-earth\Snakefile", line 592, in __rule_simplify_network
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file C:\Users\le_re\Documents\PyPSA\pypsa-earth\Snakefile, line 592:
Command 'C:/Users/le_re/mambaforge/envs/pypsa-earth/python.exe "C:\Users\le_re\Documents\PyPSA\pypsa-earth\.snakemake\scripts\tmpypqh1c2l.simplify_network.py"' returned non-zero exit status 1.
  File "C:\Users\le_re\Documents\PyPSA\pypsa-earth\Snakefile", line 592, in __rule_simplify_network
  File "C:\Users\le_re\mambaforge\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
Removing output files of failed job simplify_network since they might be corrupted:
resources/bus_regions/connection_costs_s.csv
WARNING:snakemake.logging:Removing output files of failed job simplify_network since they might be corrupted:
resources/bus_regions/connection_costs_s.csv
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-10-24T104521.772466.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake\log\2023-10-24T104521.772466.snakemake.log

[ekatef]

Hello @dhoop-iidev! Thanks a lot for reporting.

As a quick workaround, you may set p_threshold_merge_isolated in your config.yaml to false instead of 300.

Not sure about the reason of this issue. As a double-check: have you cleaned-up results of the previous runs before building the model for Argentina? Could you also please share your config settings?

I'll try to reproduce the issue locally in meantime.

[dhoop-iidev]

Hello @ekatef ! Thanks for your quick answer. Here is my original config file. I'll try to set it to clean it all and rebuild it from scratch, but I don't think that's the issue. Thanks for the workaround, though I'm afraid I could face the issue during the cluster_network phase. config.txt

[ekatef]

@dhoop-iidev thanks for the config!

Unfortunately, I can't reproduce the issue you reported. So, the usual trouble-shooting should hopefully resolve the issue.

As a suggestion, could you please remove all the output and re-run the simulation setting download_osm_data: true? It may help to fix effects on the data mismatch which sometime happen due to mixing outputs from different runs.

[dhoop-iidev]

Alright I'll keep you posted if I ever find the solution

[Javier-Andres-Gonzalez]

Hi dhoop-iidev

Did you find the solution to the problem ?

I ask because I have the same problem.

I look forward to your comments

Thanks

[dhoop-iidev]

Hello @Javier-Andres-Gonzalez, Not yet, I'll do a deep dive into the equations to try to understand where it goes wrong. But it's nice to see that I'm not alone, it means that there could be a more global issue. Tell me if you found some solutions on your part ! Saludos

[Javier-Andres-Gonzalez]

Hello @dhoop-iidev

If I can fix the problem, I will share the solution!

I have spent a long time trying to find the problem, but still nothing.

Do you speak spanish?

[dhoop-iidev]

Yes I live in Uruguay :) But let's continue in English so that the rest of the community can understand

[dhoop-iidev]

Are you also trying to model Argentina or another country ?

[Javier-Andres-Gonzalez]

I live in Argentina.

I'm trying to follow the tutorial: (https://pypsa-eur.readthedocs.io/en/latest/tutorial.html)

I am running this command: ( Error)

snakemake -call results/test-elec/networks/elec_s_6_ec_lcopt_Co2L-24H.nc --configfile config/test/config.electricity.yaml

[ekatef]

Hello @Javier-Andres-Gonzalez, as discussed, the issue has been connected with different design of PyPSA-Earth and PyPSA-Europe. Since you are interested in modeling Argentina, you need pypsa-earth, which tutorial is available here https://pypsa-earth.readthedocs.io/en/latest/short_tutorial.html

Thanks for reaching out and looking forward to hearing about your progress.

[ekatef]

Hello @dhoop-iidev!

Do you have any updates regarding the issue you faced with simplify_network?

A short trouble-shooting list which may resolve many problems with running the modeling workflow (have added a point to our documentation wish-list on that): 1) remove all the output of the workflow (the whole network folder and content of resouces folder, except natura.tiff); 2) update you local pypsa-earth repository with git pull; 3) update pypsa-earth environment with mamba env update --file envs/environment.yaml --prune

[dhoop-iidev]

Thanks @ekatef, I think I've already done that in some order, but I'll run it again and let you know. If not I'll reinstall the whole thing from scratch.

[dhoop-iidev]

Alright I delete everything and cloned pypsa earth again from scratch and this time it worked. I don't know what the error was.

[ekatef]

Alright I delete everything and cloned pypsa earth again from scratch and this time it worked. I don't know what the error was.

@dhoop-iidev Thanks a lot for notifying! Congratulations with getting the model work 🙂