PSLmodels / OG-Core

An overlapping generations model framework for evaluating fiscal policies.
https://pslmodels.github.io/OG-Core/
Creative Commons Zero v1.0 Universal
65 stars 111 forks source link

Update `demographics.py` to use the new UN API format #931

Closed SeaCelo closed 2 months ago

SeaCelo commented 3 months ago

The API for the UN population data used in demographics.py is being updated to use bearer tokens. The system is still not live, but the new format will look like this (requires a value for your_token_here):

-- USING PYTHON HTTP.CLIENT
import http.client
conn = http.client.HTTPSConnection("population.un.org")
payload = ''
headers = {
  'Authorization': 'Bearer your_token_here'
}
conn.request("GET", "/dataportalapi/api/v1/data/indicators/1/locations/688/start/2001/end/2005?pagingInHeader=false&format=json", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
--
OR
--- USING PYTHON REQUESTS ---
import requests
url = https://population.un.org/dataportalapi/api/v1/data/indicators/1/locations/688/start/2001/end/2005?pagingInHeader=false&format=json
payload = {}
headers = {
  'Authorization': 'Bearer your_token_here'
}
response = requests.request("GET", url, headers=headers, data=payload)
print(response.text)
---

As soon as the API is live, and the method to generate the bearer token is published, I will update and test the code.

jdebacker commented 3 months ago

Great - thanks for keeping up with this @SeaCelo !

jdebacker commented 3 months ago

@rickecon and @SeaCelo Here's an idea: Since the UN data files are quite small (150KB gets you almost 100 years of the population distribution by age) and since relatively few users will be able to get a UN API token, what if we create repo in EAPD-DRB calls "UN Population Data". This will have a README briefly explaining the files and then directories for each OG-XXX model, which contain 5 CSV files with the relevant data for each country calibration.

We can then change the ogcore.demographics.get_un_data() to have its current functionality if someone has the API token, but if not, it reads the relevant data files from the "UN Population Data" repo.

Some advantages of this: 1) It centralizes the storage of data so when there are changes to the UN WPP data (e.g., they add another forecast year), there's just one place to update them. 2) The ogcore.demographics can remain largely the same.

Let me know your thoughts.

jdebacker commented 2 months ago

Addressed in PR #936 using the methods suggested in the previous comment.