Update calibration - Githubissues

jdebacker commented 1 year ago

This PR updates the OG-USA calibration to move it forward from 2021 as a base year to 2023.

Some bugs are also fixed along the way.

codecov-commenter commented 12 months ago

Codecov Report

Attention: 46 lines in your changes are missing coverage. Please review.

Comparison is base (422c7b9) 78.87% compared to head (39db59b) 73.76%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #73 +/- ## ========================================== - Coverage 78.87% 73.76% -5.11% ========================================== Files 21 24 +3 Lines 1382 1616 +234 ========================================== + Hits 1090 1192 +102 - Misses 292 424 +132 ``` | [Flag](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | `73.76% <79.27%> (-5.11%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | Coverage Δ | | |---|---|---| | [cs-config/cs\_config/functions.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-Y3MtY29uZmlnL2NzX2NvbmZpZy9mdW5jdGlvbnMucHk=) | `19.66% <ø> (ø)` | | | [ogusa/calibrate.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1c2EvY2FsaWJyYXRlLnB5) | `38.51% <100.00%> (ø)` | | | [ogusa/estimate\_beta\_j.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1c2EvZXN0aW1hdGVfYmV0YV9qLnB5) | `19.73% <100.00%> (+1.07%)` | :arrow_up: | | [ogusa/macro\_params.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1c2EvbWFjcm9fcGFyYW1zLnB5) | `100.00% <100.00%> (ø)` | | | [ogusa/psid\_data\_setup.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1c2EvcHNpZF9kYXRhX3NldHVwLnB5) | `100.00% <100.00%> (ø)` | | | [ogusa/utils.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1c2EvdXRpbHMucHk=) | `100.00% <100.00%> (ø)` | | | [setup.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-c2V0dXAucHk=) | `0.00% <ø> (ø)` | | | [tests/test\_calibrate.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-dGVzdHMvdGVzdF9jYWxpYnJhdGUucHk=) | `100.00% <ø> (ø)` | | | [tests/test\_demographics.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-dGVzdHMvdGVzdF9kZW1vZ3JhcGhpY3MucHk=) | `100.00% <100.00%> (ø)` | | | [tests/test\_get\_micro\_data.py](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-dGVzdHMvdGVzdF9nZXRfbWljcm9fZGF0YS5weQ==) | `83.95% <ø> (ø)` | | | ... and [8 more](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/73/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jdebacker commented 12 months ago

As part of this update to the calibration of OG-USA, I'd like to move to using the UN Population Prospects data, as we are using in other country models.

Here's a comparison of the "old" way of calibrating US demographics (as is done in the master branch here) and the "new" way (using UN projections):

UN_pop_comparision UN_mortality_comparison UN_fertility_comparion imm_rates_comparison

The overall population and mortality rates look quite similar. There's a big rightward shift in the fertility rates, which could be because the "old" method used fertility rates from 2005, and the UN data are representing 2023, 18 years later. The rightward shift would be consistent with the increasing age at which parents have their first child over these years.

What is more difficult to explain is the sharp increase in immigration rates at the higher ages in the "new" method. This warrants some more exploration.

jdebacker commented 12 months ago

I was using pre-COVID years for estimating immigration rates, but one needs to be consistent to use years that end in the same year the other data start in (e.g., 2023 in the examples above). With that fix, I find immigration rates that look much better: UN_comparison_updated_immigration

jdebacker commented 11 months ago

SS currently solving with baseline parameterization, but nan's in solution to time path. Will find cause and fix...

rickecon commented 11 months ago

@jdebacker. I need you to make sure your main branch is updated and merge those changes into this branch. Otherwise, I can't get it onto my local git repository.

jdebacker commented 11 months ago

@rickecon Sorry about that - just synced and pushed changes to this branch.

jdebacker commented 11 months ago

FYI, the issue I'm having is that I am able to solve the baseline parameterization of the model, estimate tax functions for the reform policy and update the parameterization, but they as the reform parameterization enters the SS, things freeze up. Here's the last bit of what I see in my terminal:

making dir:  /Users/jason.debacker/repos/OG-USA/examples/OG-USA-Example/OUTPUT_REFORM/SS
making dir:  /Users/jason.debacker/repos/OG-USA/examples/OG-USA-Example/OUTPUT_REFORM/TPI
In runner, baseline is  False
Shape HH =  (80, 10) 80 10
Shape firm =  (1,) 1
Using previous solutions for SS
2023-08-16 15:14:31,355 - tornado.application - ERROR - Exception in callback <bound method SystemMonitor.update of <SystemMonitor: cpu: 0 memory: 1535 MB fds: 122>>
Traceback (most recent call last):
  File "/Users/jason.debacker/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/tornado/ioloop.py", line 919, in _run
    val = self.callback()
  File "/Users/jason.debacker/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/distributed/system_monitor.py", line 160, in update
    net_ioc = psutil.net_io_counters()
  File "/Users/jason.debacker/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/psutil/__init__.py", line 2119, in net_io_counters
    rawdict = _psplatform.net_io_counters()
OSError: [Errno 12] Cannot allocate memory

rickecon commented 11 months ago

@jdebacker. I am pretty sure this is a package conflict issue, and I think that means this has to be the addition of openssl=1.1.1 in environment.yml. The error in the GH Action CI tests is in the build section installing the environment. I did the following steps to replicate this issue locally.

I pulled your branch to my local machine
I deleted my ogusa-dev conda environment.
I updated conda conda update conda, then I updated anaconda conda update anaconda
I created the ogusa-dev conda environment from your branch conda env create -f environment.yml

My conda environment creation stalled for 1.5 hours on the "Solving environment" step.

Just to make sure, I followed the same steps in my master branch of OG-USA. The ogusa-dev conda environment was successfully created and the run_og_usa.py run script ran the baseline steady state and transition path and reform steady state and transition path.

BASELINE, Stead-state equilibrium output

SS using initial guess factors for r and TR of 1.0 and 1.0 , respectively.
SS using initial guess factors for r and TR of 0.95 and 1.0 , respectively.
SS debt =  1.4463530028797453 0.018388712512252093
IO:  (1, 1) , C:  (1,)
Steady state government spending is negative to satisfy budget
Checking constraints on capital, labor, and consumption.
    There were no violations of the constraints on labor  supply.
    There were no violations of the constraints on  consumption.
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-USA/examples/OG-USA-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

BASELINE, Transition path equilibrium output (48 min, 1.4 sec)

Maximum debt ratio:  6.4516723269879845
w diff:  2.2894147022611833e-07 -8.884648166063158e-07
r diff:  1.321677125104559e-07 -3.388333799647647e-08
r_p diff:  6.086543469230321e-08 -4.8614574504890484e-08
p_m diff:  0.0 0.0
BQ diff:  1.3310186462375806e-07 -2.298148904511521e-08
TR diff:  6.443340344608472e-08 -4.7192884609220664e-07
Iteration: 24
    Distance: 9.09282661278523e-06
Max absolute value resource constraint error: 7.261091993365776e-07
Checking time path for violations of constraints.
Max Euler error, savings:  6.7195138342412974e-12
Max Euler error labor supply:  2.8248514638562483e-12
Time path iteration complete.
It took 2931.8420736789703 seconds to get that part done.
run time =  2931.8421618938446

REFORM, Stead-state equilibrium output

SS debt =  1.267790978821387 0.01611850204532088
IO:  (1, 1) , C:  (1,)
Steady state government spending is negative to satisfy budget
Checking constraints on capital, labor, and consumption.
    There were no violations of the constraints on labor  supply.
    There were no violations of the constraints on  consumption.
/opt/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/ogcore/SS.py:1410: UserWarning: Warning: The combination of the tax policy you specified and your target debt-to-GDP ratio results in an infeasible amount of government spending in order to close the budget (i.e., G < 0)
  warnings.warn(
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-USA/examples/OG-USA-Example/OUTPUT_REFORM/SS/SS_vars.pkl

REFORM, Transition path equilibrium output (45 min, 1.4 sec)

Maximum debt ratio:  3.161959684488428
w diff:  1.1565875330799003e-06 -9.69099389180883e-07
r diff:  1.6580871832483357e-07 -1.558311004054569e-07
r_p diff:  1.0626047068640343e-07 -1.059256963739652e-07
p_m diff:  0.0 0.0
BQ diff:  1.0146416001485803e-07 -8.84533225518902e-08
TR diff:  2.048346303823445e-07 -1.9448447746012176e-07
Iteration: 22
    Distance: 6.881648356133613e-06
Max absolute value resource constraint error: 3.418417115416683e-07
Checking time path for violations of constraints.
Max Euler error, savings:  2.5932034297682094e-12
Max Euler error labor supply:  1.304734098539484e-12
Time path iteration complete.
It took 2751.0054788589478 seconds to get that part done.
run time =  2751.0055270195007
Percentage changes in aggregates: Year                    Variable   2021   2022  ...   2030  2021-2030     SS
0                    GDP ($Y_t$)  -7.95  -7.95  ...  -7.12      -7.80 -12.35
1            Consumption ($C_t$)  -8.28  -7.16  ... -13.64     -11.11 -19.34
2          Capital Stock ($K_t$)  -1.36  -2.43  ...  -5.08      -4.30 -21.82
3                  Labor ($L_t$) -11.32 -10.79  ...  -8.19      -9.57  -6.78
4     Real interest rate ($r_t$) -18.54 -15.94  ...  -7.51     -10.61  20.29
5                      Wage rate   3.80   3.18  ...   1.18       1.99  -5.97

[6 rows x 13 columns]

jdebacker commented 11 months ago

@rickecon Writes:

@jdebacker. I am pretty sure this is a package conflict issue, and I think that means this has to be the addition of openssl=1.1.1 in environment.yml.

But I have a built environment with openssl=1.1.1... and I am able to solve the baseline once. There are a bunch of other changes in this PR -- to the example script and to the parameterizations.

If you are having trouble building the environment in this branch, maybe use the one you built from the master branch and then downgrade to openssl 1.1.1. And then see if you have errors similar to me.

rickecon commented 11 months ago

In my ogusa-dev conda environment freshly installed from master, I did a conda install openssl=1.1.1. This is a big downgrade as the unconstrained version that installs in my ogusa-dev is version 3.1.2. The conda downgrade runs for 40 minutes, looking for a combination without conflicts, then fails. The traceback is huge, but it is basically lots of iterations of package conflicts for each package. But it finishes without being able to downgrade openssl. I think we have to either find a way to access the UN portal with a newer version of openssl, or we have to hold off on this PR. @SeaCelo, what do you think about the UN Data Portal upgrading to allow openssl>=3.1.2?

rickecon commented 11 months ago

@jdebacker @SeaCelo. I took a long shot and relaxed the openssl dependency in environment.yml and was able to create the ogusa-dev conda environment. In this new environment, the openssl version on my machine is v3.1.2. But it still breaks when accessing the UN Data Portal API.

I ran the run_og_usa.py run script from your branch, and it solved the baseline steady-state and transition path (see below). But then it broke at the reform, after the tax functions, when it went to access the UN demographic data. See traceback below. I am going to try adding some of the version limitations that we added to OG-ZAF's environment.yml file.

BASELINE, Steady-state equilibrium output

SS using initial guess factors for r and TR of 1.0 and 1.0 , respectively.
SS debt =  1.2964441119522576 0.008458992736776541
IO:  (1, 1) , C:  (1,)
Steady state government spending is negative to satisfy budget
Checking constraints on capital, labor, and consumption.
    There were no violations of the constraints on labor  supply.
    There were no violations of the constraints on  consumption.
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-USA/examples/OG-USA-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

BASELINE, Transition path equilibrium output (53 min, 0.0 sec)

Maximum debt ratio:  2.0001056114786286
w diff:  6.156471861196167e-07 -7.861589268465252e-08
r diff:  1.113320424739328e-08 -9.295587860558285e-08
r_p diff:  9.964607652901414e-09 -6.034545763783372e-08
p_m diff:  0.0 0.0
BQ diff:  7.13387282927358e-08 -9.346743323063178e-08
TR diff:  5.941657506280951e-08 -4.903460550098604e-08
Iteration: 23
    Distance: 6.977706548003614e-06
2023-08-17 12:03:07,153 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-08-17 12:03:07,525 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-08-17 12:03:07,894 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
Max absolute value resource constraint error: 2.2207033976984203e-07
Checking time path for violations of constraints.
Max Euler error, savings:  3.339550858072471e-12
Max Euler error labor supply:  1.0229594948896192e-12
Time path iteration complete.
It took 3182.9096410274506 seconds to get that part done.
run time =  3182.9097158908844

REFORM, Error traceback

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-USA/./examples/run_og_usa.py", line 144, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-USA/./examples/run_og_usa.py", line 87, in main
    c2 = Calibration(
  File "/Users/richardevans/Docs/Economics/OSE/OG-USA/ogusa/calibrate.py", line 60, in __init__
    self.demographic_params = demographics.get_pop_objs(
  File "/Users/richardevans/Docs/Economics/OSE/OG-USA/ogusa/demographics.py", line 437, in get_pop_objs
    fert_rates = get_fert(E + S, min_age, max_age)
  File "/Users/richardevans/Docs/Economics/OSE/OG-USA/ogusa/demographics.py", line 115, in get_fert
    df = get_un_data("68")
  File "/Users/richardevans/Docs/Economics/OSE/OG-USA/ogusa/demographics.py", line 65, in get_un_data
    response = requests.get(target)
  File "/opt/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/anaconda3/envs/ogusa-dev/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='population.un.org', port=443): Max retries exceeded with url: /dataportalapi/api/v1/data/indicators/68/locations/840/start/2023/end/2023 (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1007)')))

SeaCelo commented 11 months ago

@rickecon @jdebacker I've reached out for an update on adding SSLv3 support to the web server. I'll keep you updated.

jdebacker commented 11 months ago

@SeaCelo - thank you!

I think my issue is something else, but it would be great to allow more recent SSL protocol to access these data.

SeaCelo commented 11 months ago

@rickecon @jdebacker I've created a PR (https://github.com/EAPD-DRB/OG-ZAF/pull/42) that should fix the problem with openssl. Please test. If it works, we can remove openssl=1.1.1 as a dependency.

SeaCelo commented 10 months ago

@rickecon @jdebacker I've created a PR (EAPD-DRB/OG-ZAF#42) that should fix the problem with openssl. Please test. If it works, we can remove openssl=1.1.1 as a dependency.

@rickecon @jdebacker The solution works and we no longer need the older openssl that is causing the conflicts.

On this PR it requires defining the function in utils.py and calling it when accessing the UN Data Portals in demographics.py and macro_params.py. Then removing the requirement in environments.yml. I can submit the changes to this PR or one of you can do it.

jdebacker commented 9 months ago

I am able to solve the baseline and reform in the run_og_usa.py script when using the GS functional form for tax functions.

Note two oddities when working with the example script in this PR:

The model did not solve with the DEP functional form, failing to find a SS eq'm in the baseline. When plotting the DEP functions by age (for 2023, the first year of the model), there is some variation age to age in the functions' shape that could be a cause:
When trying to estimate the mono and mono2D functional form for the tax functions, there were failures in the estimation (e.g., no minimum found).

Despite this, I still feel that this PR is worth merging as makes important updates to the parameters and there are no changes to how the tax functions are handled, so it ought to be the case that the oddities with the DEP and mono functions are related to the Tax-Calculator output (although in plotting the full taxcalc output and in summary statistics, I could not find any particular patterns that would point to significant differences from prior versions).

@rickecon thoughts?

I'll also try to address why the environment is not building for the tests on GH...

jdebacker commented 8 months ago

@rickecon I made the changes we talked about today (remove references to surv_rate and make rho 2D) and think this PR should be ready to merge.

rickecon commented 8 months ago

@jdebacker. I just submitted a PR to your branch that fixes some errors stemming from demographics.py. I also changed the build section of build_and_test.yml by adding the Mambaforge approach in the hopes that it would help the CI progress through that section better.

Some unresolved issues with this PR are:

The documentation does not build. We should fix this before moving on.
The demographics.py plotting functions in get_fert(), get_mort(), and get_imm_rates() doesn't work. We can move forward temporarily without this, and we can fix it in a future PR.

jdebacker commented 8 months ago

@rickecon Thanks for your PR to this branch. It's useful to have booleans to turn on plotting capabilities (though I wasn't having any issues with them).

GH Actions still having issues, but I'm finding that all tests pass locally:

(ogusa-dev) jason.debacker@JDEBACKER-7 OG-USA % pytest
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: /Users/jason.debacker/repos/OG-USA
configfile: pytest.ini
testpaths: ogusa/tests
plugins: xdist-3.3.1, pep8-1.0.6
collected 45 items

ogusa/tests/test_calibrate.py ....                                       [  8%]
ogusa/tests/test_demographics.py ........                                [ 26%]
ogusa/tests/test_get_micro_data.py ..............                        [ 57%]
ogusa/tests/test_income.py ............                                  [ 84%]
ogusa/tests/test_psid_data_setup.py ...                                  [ 91%]
ogusa/tests/test_run_example.py .                                        [ 93%]
ogusa/tests/test_utils.py .                                              [ 95%]
ogusa/tests/test_wealth.py ..                                            [100%]

================= 45 passed, 95 warnings in 1600.82s (0:26:40) =================

rickecon commented 8 months ago

@jdebacker. What should we do here? I think we need at least some tests to pass on the GitHub CI before merging. And GitHub Actions are going really slowly. But there is one test running right now. I would love to see that one go through and pass.

rickecon commented 8 months ago

@jdebacker. Thanks for this PR. I am merging this now. We couldn't get the GH Actions Windows tests to run (Python 3.9 and Python 3.10), but all the other tests ran and passed. Hopefully we can address why the GH Actions are behaving so erratically.

PSLmodels / OG-USA

Update calibration #73

Codecov Report