search

PyPSA / pypsa-eur

PyPSA-Eur: A Sector-Coupled Open Optimisation Model of the European Energy System
https://pypsa-eur.readthedocs.io/
343 stars 242 forks source link

Add optional rule for updating network topology from entsoe map #55

Closed coroa closed 3 months ago

coroa commented 5 years ago

Must be optional, since it is based on GridKit and requires a working Postgres database with PostGIS. Based on https://github.com/PyPSA/GridKit/tree/master/entsoe and https://rustyb.github.io/ee_transmap/.

nworbmot commented 5 years ago

Need to also update heuristic for reading line tags (e.g. multiple circuit types).

coroa commented 5 years ago

To make some headway here, there are 4 steps this entails:

  1. Getting the raw geojson data from their new mapbox endpoint
  2. Importing them into a postgres database with the postgis extension
  3. Running Gridkit on them
  4. Exporting them

All steps have to be run on the https://github.com/PyPSA/GridKit repo in the entsoe directory.

Step 1: Getting the raw GeoJSON data

This is in principle accomplished by running the download.sh script. You can remove the two rusty handles from the while loop, those are the previous datasets (as used in pypsa-eur, atm). One of the entsoe.* files will contain buses, one will contain line segments.

The problem is, that the underlying vt-geojson library hasn't received any love in the last year and the rapidly changing javascript ecosystem has long evolved away.

I found it is possible to get it to run anyway with the help of an ancient docker image for node:

docker run -it -v $PWD:/app:rw -w /app node:8.16.0-alpine sh download.sh

Steps 2-4: Import, running and export

Are governed in order by the shell-scripts import.sh, run.sh and export.sh (creative isn't it).

These depend on a postgres database server with the PostGIS extension and a python installation with several packages. It's probably possible with the correct docker image again, but previously I just installed the postgres things with apt.

Once postgres is running, you must create a new database (maybe install the PostGIS extension there?) and set the environment variables PG_USER, PG_PASSWORD and PG_DATABASE in your shell.

The python environment (don't know whether python3 works or you'll have to go back to 2.7) should contain:

psycopg2
geopandas
fiona
geojson

Then, you should (brace for impact) be able to execute the shell scripts in order and finally get all the csv files which you need for pypsa-eur.

Note that the ids and oids differ from the version in pypsa now, so that you will have to re-write the data/parameter_corrections.yaml file, maybe what is in there has been already fixed in the upstream map, though.

I'll try watching this thread, if you need further instructions.

pz-max commented 2 years ago

We discussed this today. We won't add an optional rule but aim to document the process how to update the network. @martacki is speaking to @coroa to get some information on the process (if understood correctly)

fneum commented 3 months ago

closing as we will phase-out the use of the ENTSO-E map in favor of OSM data (#1079)