Prepare the MERGE

[davide-f]

Copying from https://github.com/pypsa-meets-earth/pypsa-earth-sec/pull/308

Changes proposed in this Pull Request

This PR should clean the scripts, enhance the readability and overall architecture of the model preparing for a future model merge with the power sector model. All open PRs should be reviewed and merged before this one.

Subtasks

This PR consists of sevaral tasks

• [ ] 1. Prepare the config files.
  • [ ] Clean the config file and remove any duplication.
  • [x] Align with PyPSA-Earth -> #313 and what remains can be done later
  • [x] Use a single config file -> #313
• [ ] 2. Improve the script prepare_sector_network.py.
  • [ ] Enhance code quality and architecture. -> optional, not spend too much time here for the merge
  • [ ] Add sufficient documentation.
• [ ] 3. Revise the heat sector.
  • [ ] Check the heat scripts. -> secondary priority
• [ ] 4. Prepare the snakefile.
  • [ ] #330
  • [ ] Add modularity -> TO DO AFTER the merge
  • [ ] Enable scenario analysis and link it with PyPSA-Earth -> second priority
• [ ] 5. General style improvement
  • [ ] Drop warnings from scripts #319
  • [ ] Remove duplication of helper information (using sys.append in the helpers to link scripts from the main repo for example) -> secondary priority, thanks to https://github.com/pypsa-meets-earth/pypsa-earth/pull/1029 and potentially finalized during the merge
• [ ] 6. Other functionalities
  • [ ] Support last environment and pypsa version, by adopting linopy. PyPSA-Earth has a ready PR that supports it and it may be interesting to have this too

[finozzifa]

Use a single config file --> https://github.com/pypsa-meets-earth/pypsa-earth-sec/pull/313

[finozzifa]

Hi @davide-f, what is it meant exactly with Prepare the snakefile, align with PyPSA-Earth Thanks : )

[davide-f]

Macrotopics

Usability

• compatibility with previous repos

Procedure

• define procedure to avoid conflicts
  • avoid files in the scripts folder to have the same name across repos
  • do not have file "snakefile" in both repos
  • make sure no other file has the same path and name
• define the repo structure [the -eur one?]
  • have a scripts folder only or more? 1 only is easy but overwhelming; more may be tricky from a coding perspective
  • Maybe define smart naming conventions: e.g. retrievepower/sec... Add these only if needed
• restructure the repos to make sure to avoid git merge conflicts
• overlapping rules need to be sorted out
• move issues across repos

1. config files
2. snakefile modularity
3. styling convention of python (snakemake params and scenario runs)
4. structure of the repo
   • we keep 1 script folder only
   • do we want the networks folder?
5. how to make sure to satisfy usability?
   • make sure that the commands for both CIs work the same
6. need to revise the documentation folder accordingly

COMMUNICATE TO THE COMMUNITY:

• we go for the merge
• in the next weeks, we will be preparing the repos
• need to close PRs before merge
• once we start the merge, no more merge and it will break existing PRs

Past experience

In -eur there have been misalignment between the repos to be fixed. A lot of brute force.

First, the two packages have been merged (brute force) and then let's see. Most snakefile rules were diverging, thus lot of rearrangement, done during the merge-> the snakefile has been modularized into multiple .smk files. The config: large merge issue: append entries first and then consolidated. Done lots of brute force. [3 intense days] Not much preparation

Suggestion: make sure that the two git repositories are on the same state: use the latest version.

Concrete steps

1. config files
2. snakefile modularity: secondary priority
3. styling convention of python:
   • drop deprecations
   • update to linopy?
   • snakemake params [not necessary but maybe nice to have]
   • scenario runs [third priority]
4. structure of the repo
   • we keep 1 script folder only
   • do we want the networks folder? -> ask to the community
5. how to make sure to satisfy usability?
   • make sure that the commands for both CIs work the same
6. need to revise the documentation folder accordingly

Timeline

• low availability by Hazem
• fabian, from 2 weeks may have availability

2-4 weeks preparation weekly update during the -sec model 4-8 weeks target for merge success

[ekatef]

In part of linopy integration: we have a PR opened in pypsa-eath. No problem to keep it opened to after merge. But also happy to support integration of it into pypsa-earth-sec before merge.