pypsa-meets-earth / pypsa-earth-sec

GNU General Public License v3.0
20 stars 17 forks source link

Dynamically download urban_percent #262

Closed davide-f closed 8 months ago

davide-f commented 8 months ago

Checklist

Describe the Bug

Data in file urban_percent are limited to few countries that limit its usage at africa or world scale. That information (percentage of urban population) is accessible from databank worldbank portal (along with a ton of other informations). See url: https://databank.worldbank.org/source/world-development-indicators Database: World Development Indicators Serie: "Urban population (% of total population)

This link: https://databank.worldbank.org/AjaxDownload/FileDownloadHandler.ashx?filename=P_fd6c7038-d48b-479f-8411-bc1795fa6adb.zip&filetype=CSV&language=en&displayfile=P_Data_Extract_From_World_Development_Indicators.zip temporarily downloads part of the data, not sure how stable that is. It may be good to investigate some APIs of the portal

From here: https://datatopics.worldbank.org/world-development-indicators/ the entire dataset can be downloaded

From here the specific serie can be seen: https://databank.worldbank.org/reports.aspx?dsid=2&series=SP.URB.TOTL.IN.ZS

There may be other files that have a similar problem, such as hydrogen_salt_vavern, export_ports (it makes sense), tbd

Error Message

If applicable, paste any terminal output to help illustrating your problem. In some cases it may also be useful to share your list of installed packages: conda list.

KeyError: 'AO'
davide-f commented 8 months ago

FYI @doneachh

davide-f commented 8 months ago

There is wbgapi that can be used to download data of databank. https://blogs.worldbank.org/opendata/introducing-wbgapi-new-python-package-accessing-world-bank-data

If we use various series, then it may be worth using, otherwise we could investigate its functionality and reproduce an effect if short. It exploits requests, so it shouldn't be a big deal bit to be verified xD

davide-f commented 8 months ago

Found! :D https://api.worldbank.org/v2/country/all/indicator/SP.URB.TOTL.IN.ZSsource=2&downloadformat=csv

more at: https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structures