search

pypsa-meets-earth / pypsa-earth-sec

GNU General Public License v3.0
19 stars 16 forks source link

Fix workflow for running Africa #285

Closed davide-f closed 1 month ago

davide-f commented 4 months ago

Closes # (if applicable).

Changes proposed in this Pull Request

I openned this PR to describe current efforts in enabling the workflow to run for Africa. This PR integrates #284

Checklist

davide-f commented 3 months ago

@doneachh @Eddy-JV @hazemakhalek @energyLS this PR is ready for a first revision of the procedure and highlights. But do not merge yet

davide-f commented 3 months ago

FYI this PR has run for Africa locally with the custom data stored in the parallel PR opened here, but the model is infeasible. Numbers need checking

davide-f commented 3 months ago

@hazemakhalek unfortunately the use of 40000TWh of biomass potential didn't solve the issue

doneachh commented 3 months ago

@davide-f could you summarize for me, which tasks are open at this point for getting serious africa modeling results?

hazemakhalek commented 3 months ago

@hazemakhalek unfortunately the use of 40000TWh of biomass potential didn't solve the issue

Out of experience, 90% of the cases the problem is either biomass potential, oil potential or co2 emission limit

hazemakhalek commented 3 months ago

The PR looks alright to me, I am not able to run it though as I'm currently out of office. All the logic seem alright to me so if you tested it @davide-f and the CI passes I think we can merge it

davide-f commented 3 months ago

The PR looks alright to me, I am not able to run it though as I'm currently out of office. All the logic seem alright to me so if you tested it @davide-f and the CI passes I think we can merge it

Great to hear! and many thanks :)

I'll wait merging it till at least this is merged. That leads to some little issues in the execution that can apply to -sec. I'm locally testing it for Africa (-sec version) and World (only power).

davide-f commented 3 months ago

This has worked for Africa, but with mathematical unfeasibility though; Ready to review and finalize :)

davide-f commented 3 months ago

Many thanks! @doneachh ! One last update to be merged after review in pypsa-earth though: https://github.com/pypsa-meets-earth/pypsa-earth/pull/992

davide-f commented 3 months ago

Moreover, this PR needs some checks: for MA it doesn't work well. I'm investigating

davide-f commented 3 months ago

@davide-f could you summarize for me, which tasks are open at this point for getting serious africa modeling results?

  • model doesnt have enough biomass potential
  • Missing H2-Buses for GHA
  • Something else as well?

Hello :D Sorry my late reply, I didn't see your review but I've answered all your comments.

Regarding the rest:

Overall, I feel like there is the need for some debugging for larger regions. I'd recommend to start "easy": rather then modelling Africa as a whole, maybe start with a power pool, then follow to other power pools and finally the whole model. That is to get confidence of the results at regional level. We need to check how the model performs with multiple countries: in this PR I aimed to fix the limitations I saw in prepare_sector_network that was coded to represent a single country, but there may be other limitations in the whole code. That needs some checking

davide-f commented 3 months ago

For me this PR is ready to go. I'd recommend somebody to rerun it locally as well on MA to have a crosscheck.

davide-f commented 1 month ago

I think I discovered the issue and the problem relates to isolated nodes. What I did is several testing. The latest one that helped me achieve this insight is:

  1. run MA only with 20 nodes -> working
  2. run MA+DZ with 20 nodes -> failing due to numerical issues
  3. run MA+DZ with 15 nodes and varying CO2 limits -> failing
  4. run DZ only with 10 nodes -> failing
  5. checked the DZ-only network that was supposed to have only 10 nodes and instead it has 36

I see 2 options:

  1. merge this PR
  2. first rerun a MA scenario again comparing original and new approach and merge it
doneachh commented 1 month ago

I think I discovered the issue and the problem relates to isolated nodes. What I did is several testing. The latest one that helped me achieve this insight is:

  1. run MA only with 20 nodes -> working
  2. run MA+DZ with 20 nodes -> failing due to numerical issues
  3. run MA+DZ with 15 nodes and varying CO2 limits -> failing
  4. run DZ only with 10 nodes -> failing
  5. checked the DZ-only network that was supposed to have only 10 nodes and instead it has 36

I see 2 options:

  1. merge this PR
  2. first rerun a MA scenario again comparing original and new approach and merge it

@davide-f Great to hear that you have found the issue! :) Do you already know why DZ has 36 nodes instead of the 10 specified in the config?

What do you mean with original and new approach?

From my side we can merge this PR! :)

doneachh commented 1 month ago

@davide-f Code looks good to me! You can merge it, if you want :)

davide-f commented 1 month ago

Thanks @doneachh !

@hazemakhalek following the discussion. To verify no additional issue is added here, this PR has been further tested for MA using upstream main and this PR. The objective value changes slightly, about 1-2%, the reasons are the following:

  1. this PR fixes a bug in the heat demand that was previously omitted
  2. the update of the submodule has changes slightly something

If that's fine for you, we can merge. I'm rerunning the PR to check if it still works for Africa given the latest changes but it contains most improvements needed. Minor issues may be left

doneachh commented 1 month ago

@davide-f Fine for me!

davide-f commented 1 month ago

Africa has successfully executed locally :) merging Thanks Anton! :D