Add functionality to merge each isolated bus to the backbone network

[ekatef]

Closes #852

Changes proposed in this Pull Request

Added merge_into_network( ) function to merge each isolated bus into the network following an approach of drop_isolated_nodes( ) and merge_isolated_nodes( ).

Checklist

• [x] I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• [x] I tested my contribution locally and it seems to work fine.
• [x] Code and workflow changes are sufficiently documented.
• [ ] Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• [x] Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• [ ] Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• [x] Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• [x] A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[ekatef]

Currently, the code works if a number of isolated nodes is more than one. It would be nice to generalise on a case with a single isolated nodes, preferably without need to using conditional statements.

Functionality has not been tested yet on geometries.

What would be great is to better understand requirements towards the implementation:

• do we need a voltage threshold for merge_into_network( )?
• how should be this option combines with drop of the isolated nodes and merging into a single node?

[ekatef]

Currently, fetching isolated nodes into the closest network in implemented. Tested on a CL and seems to work fine:

base.nc

elec_s.nc, current implementation

elec_s.nc, PR implementation

Red highlights buses which don't belong to the largest sub-network.

Fetching the isolated nodes keeps invariant values of the overall load and generation.

[ekatef]

What would be nice to understand, is which strategies are sensible for drop/merge/fetch isolated nodes and isolated networks and adjust configuration parameters accordingly. My feeling is that it feels like alternatives to merge isolated nodes in a single isolated node or to fetch them into network.

Another point is handling isolated fragments of the network (like those red buses on first picture above). I suspect, in this particular case they mostly get reduced to a single node and dropped or merged after. However, the may be cases when isolated networks remain as two-three "micro-networks" after simplification, and can't be merged into the "main" network with a current approach. Probably, we can find some solution for this case, as well.

@davide-f would be very grateful if you could find time for looking into it 🙂

[Edudiro]

Hi @ekatef, thank you for your response on google groups and for developing this PR. I have tested your approach and it works perfectly in the Chilean case, as you have proven above.

I am assuming the large amount of unconnected nodes from the base network comes from OSM data gaps? In that case I think following this solution is a valid approach. Otherwise, unfeasibilities might arise for a dispatch only problem.

[ekatef]

Hi @ekatef, thank you for your response on google groups and for developing this PR. I have tested your approach and it works perfectly in the Chilean case, as you have proven above.

I am assuming the large amount of unconnected nodes from the base network comes from OSM data gaps? In that case I think following this solution is a valid approach. Otherwise, unfeasibilities might arise for a dispatch only problem.

Hello @Edudiro! Great to hear that it worked for you. Thanks for giving an inspiration for pushing these developments :D

I also have a feeling that the unconnected nodes in base.nc result from some data gaps in OSM data base. At least, looking on GENI description of a national power system, it seems that there shouldn't be any reason for having these isolated power systems. However, some regional expertise may be very helpful to give a more definitive answer.

[ekatef]

@davide-f I think, the PR is ready for the first review round :)

Currently, isolated nodes are merged to the closest non-isolated ones. Further improvements are obviously possible, for example to account for the subnetwork type or implement a power threshold.

Haven't added yet configuration parameters for fetching isolated nodes, as it feels a good idea to decide on a general interfacing concept. Would be very grateful for any input on that :)

[davide-f]

As an additional comment, for a future PR, I think that we could revise in the future the behaviour of the isolated_nodes and extend to networks: drop/merge isolated networks with demand below a given threshold

[ekatef]

Sorry for the little delay, here is my feedback :D Great!

Hey @davide-f, no problem, I was aware that you were very busy. Thanks a lot for the review!

It looks like we are well aligned on the concept, and happy to introduce the changes.

Yeah, the idea of this PR as far was to focus on isolated nodes only, keeping in mind that same approach may be used to deal with isolated networks, as well.

@carlosfv92 May you be interested to work on implementation for isolated networks? Happy to discuss on align on the implementation.

[ekatef]

Testing update on a multy-country case (Congo Democratic Republic), Burundi, Rwanda

initial case

results of simplification

[davide-f]

Wow! that seems to work very well :D maybe a little too much tough... XD the actual Rwandan and Borundi networks are actually collabsed. Multi-country subneworks could be excluded from the merge probably. What do you think?

[ekatef]

Wow! that seems to work very well :D maybe a little too much tough... XD the actual Rwandan and Borundi networks are actually collabsed. Multi-country subneworks could be excluded from the merge probably. What do you think?

Absolutely agree with your assessment :D Basically, that has been testing of a limiting case which went as kind of reductio ad absurdum :)))

Looking into the regional details for this particular case, I'd say we need to answer the following questions for multi-country region applications:

1. What to do about small sub-networks which span across different countries? Agree that the best strategy seems to be just leaving such sub-networks alone.
2. Now to deal with the different scale of power systems? I mean, if there is a bid difference in the overall demand of two power systems (e.g. South Africa and Eswatini), it may be not quite handy to have the same power threshold. So, I think it may be a good idea to play with a share of the overall national demand to be used as a threshold. Let say, fetch everything with less than 5% demand of the national energy demand into the national backbone grid. Does it sound reasonable for you?

[ekatef]

Results of an updated version:

The resulted topology looks a bit odd with a new line added which spans across the whole country, parallel to the DC link.

Highlighting with green the buses of base.nc into which isolated nodes should be merged:

Highlighting after isolated network has been fetched:

So, the result may be correct in terms on buses ids, but would be great to avoid such distortions of geometry, as it can be potentially very misleading.

[ekatef]

Results of topology investigation: all the green buses should be merged into one with a circle around:

That is, the the busmap looks like follows for the backbone bus"44":

busmap.loc[busmap == "44"]
Bus
21     44
22     44
24     44
26     44
27     44
       ..
260    44
261    44
265    44
266    44

So, the transformation intention looks correct but after application of get_clustering_from_busmap the topology gets distorted for some reason.

Update: the reason of such geometry transformation is that mean is being used as an aggregation strategy for buses:

DEFAULT_BUS_STRATEGIES = dict(
    x="mean",
    y="mean",
    v_nom="max",
    v_mag_pu_max="min",
    v_mag_pu_min="max",
    generator=lambda x: "",
)

The implemention in the latest PyPSA version differs from the version we are using currently. But the idea remains the same: there is an implicit assumption that buses mapped into an aggregated node are close enough to this node. In case of weak connectivity it doesn't hold and leads to quite specific topologies.

[ekatef]

As a second thought, it may be probably not a good idea to allow fetching the nodes into a backbone network if this backbone is too far. Generally the testing looks successful. Have also replaced a relative threshold with an absolute one with the following result (with 10 MW of mean power):

The networks of smaller countries may look a bit cleaner with a relative threshold, but I'd say that keeping a consistent interface feels more important.

[ekatef]

My feeling is that it's ready to review. @davide-f I'd be very grateful if you could please look into that 🙂

[ekatef]

Hello @davide-f!

Thanks a lot for the review and apologies for taking a while to consider it. I do agree that it's worth to keep things simple as far as the target functionality is ensured. However, I have an impression that handling subnetworks is more tricky as compared with the single nodes, especially when considering a multi-countries case...

Have tried to incorporate your comments where I have seen the way to do it, and tried to explained the reasoning for the points where is hasn't been the case. Let me know please if I get your ideas right. Obviously, happy to discuss further 🙂

[ekatef]

Revised to simplify the implementation. @davide-f thank you so much for the discussion and the ideas! The current version looks definitely cleaner. Could you please review it, when you have time?

Summary of the major changes as compared with the previous iteration: 1) moved all load processing into the function which creates a buses geo-dataframe; 2) revised the threshold from to be a share of the national load instead an absolute value; 3) replaced all the functions to filter the island and backbone networks with the flags in the buses geo-dataframe.

[ekatef]

@davide-f when you have time, it would be perfect to have your review 🙂

[ekatef]

Great @ekatef :D I think the functionality is quite there and way cleaner! In the future we may reformulate but I think it is in great shape! There may not be the need to duplicate the information by subnetwork to each row of the buses, but I think it is good to go with minor changes! :) Great work and patience!

Thanks a lot for the great support and the review @davide-f!

I'd say this duplication is rather a feature than a bug ;) There is the "tidy data" concept which assumes that each row should contain a full information for a single variable. That makes life way easier even if it implies some duplication. That is, if I have get your idea properly. Happy to discuss more and adjust the implementation, if needed :)

[ekatef]

@davide-f could you please check if you have any questions on the PR, when you would have time?

It seems to be needed, and I think it's simpler to finalise it instead to explain how to test it from a fork.

[ekatef]

Great @ekatef <3 amazing contribution and sorry for the delay... I approve this PR and feel free to merge it with squash merge or normal merge as you see it more fair/clean :)

No worries and thanks a lot for the outstanding support :D