rickecon commented 1 year ago

This PR is mainly to update demographics.py to use UN data. However, I also added a bunch of other related updates. This PR does the following:

Updated demographics.py to use the UN data portal to get UK demographic data.
Updated calibrate.py to allow for demographics.
Added four .csv files to oguk/data/demographic/. This files allow for demographics.py to have the option to not download the data from the UN data portal.
Updated get_micro_data.py and test_get_micro_data.py with PolicyEngine references instead of OpenFisca.
Updated environment.yml and setup.py.
Updated run_oguk.py with more consistent specification, and updated references from openfisca to policyengine.
Small updates to .gitignore, README.md, demographics.rst, get_micro_data.rst, and tax_functions.md.
Deleted pyproject.toml which was just a reference to the black package.
Updates the CHANGELOG.md and updates the version number in setup.py.
Updates the GitHub Actions deploy_docs.yml, docs_check.yml, and build_and_test.yml.

cc: @jdebacker

rickecon commented 1 year ago

@jdebacker @nikhilwoodruff. The demographics.py module works great. I have tested all of its functions. However, I have not been able to successfully run the run_oguk.py example script. The error I am getting seems to be related to the import statement from policyengine_core.reforms import Reform. Here is the Traceback.

Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-UK/./examples/run_oguk.py", line 7, in <module>
    from policyengine_core.reforms import Reform
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/__init__.py", line 1, in <module>
    from policyengine_core.simulations import Microsimulation, Simulation
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/simulations/__init__.py", line 9, in <module>
    from .microsimulation import Microsimulation
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/simulations/microsimulation.py", line 9, in <module>
    from policyengine_core.simulations.simulation import Simulation
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/simulations/simulation.py", line 9, in <module>
    from policyengine_core import commons, periods
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/commons/__init__.py", line 1, in <module>
    from .formulas import apply_thresholds, concat, switch
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/commons/formulas.py", line 13, in <module>
    from policyengine_core.parameters.parameter_node import ParameterNode
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/parameters/__init__.py", line 15, in <module>
    from .operations import (
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/parameters/operations/__init__.py", line 1, in <module>
    from .get_parameter import get_parameter
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/parameters/operations/get_parameter.py", line 2, in <module>
    from policyengine_core.parameters.parameter_node import ParameterNode
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/parameters/parameter_node.py", line 6, in <module>
    from policyengine_core import commons, parameters, tools
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/policyengine_core/tools/__init__.py", line 6, in <module>
    import numexpr
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/numexpr/__init__.py", line 24, in <module>
    from __config__ import show as show_config, get_info
ModuleNotFoundError: No module named '__config__'

rickecon commented 1 year ago

@jdebacker. Also, the Jupyter Book documentation no longer builds. When I run jb build ./docs/book, I get the following traceback.

Running Jupyter-Book v0.13.1
Source Folder: /Users/richardevans/Docs/Economics/OSE/OG-UK/docs/book
Config Path: /Users/richardevans/Docs/Economics/OSE/OG-UK/docs/book/_config.yml
Output Path: /Users/richardevans/Docs/Economics/OSE/OG-UK/docs/book/_build/html
Running Sphinx v4.5.0
[etoc] Changing master_doc to 'content/intro/intro'
loading pickled environment... failed
failed: Can't get attribute 'NameConstant' on <module '_ast' (built-in)>
checking bibtex cache... out of date
parsing bibtex file /Users/richardevans/Docs/Economics/OSE/OG-UK/docs/book/OGUK_references.bib... parsed 2 entries
parsing bibtex file /Users/richardevans/Docs/Economics/OSE/OG-UK/docs/book/citations.bib... parsed 0 entries
myst v0.15.2: MdParserConfig(renderer='sphinx', commonmark_only=False, enable_extensions=['colon_fence', 'dollarmath', 'linkify', 'substitution', 'tasklist'], dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', disable_syntax=[], url_schemes=['mailto', 'http', 'https'], heading_anchors=None, heading_slug_func=None, html_meta=[], footnote_transition=True, substitutions=[], sub_delimiters=['{', '}'], words_per_minute=200)
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 9 source files that are out of date
updating environment: executing outdated notebooks... failed

Extension error (myst_nb.execution):
Handler <function update_execution_cache at 0x10a0e1700> for event 'env-get-outdated' threw an exception (exception: jsonschema 3.2.0 has no such extra feature 'format-nongpl')
Traceback (most recent call last):
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2760, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'format-nongpl'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/sphinx/events.py", line 94, in emit
    results.append(listener.handler(self.app, *args))
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/myst_nb/execution.py", line 76, in update_execution_cache
    _stage_and_execute(
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/myst_nb/execution.py", line 279, in _stage_and_execute
    execute_staged_nb(
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/myst_nb/execution.py", line 311, in execute_staged_nb
    executor = load_executor("basic", cache_base, logger=LOGGER)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/jupyter_cache/executors/base.py", line 78, in load_executor
    execute_cls = entry_points[0].load()
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2470, in load
    self.require(*args, **kwargs)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2493, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 803, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2762, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: jsonschema 3.2.0 has no such extra feature 'format-nongpl'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/jupyter_book/sphinx.py", line 171, in build_sphinx
    app.build(force_all, filenames)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/sphinx/application.py", line 330, in build
    self.builder.build_update()
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 286, in build_update
    self.build(to_build,
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 300, in build
    updated_docnames = set(self.read())
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 371, in read
    for docs in self.events.emit('env-get-outdated', self.env, added, changed, removed):
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/sphinx/events.py", line 102, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function update_execution_cache at 0x10a0e1700> for event 'env-get-outdated' threw an exception (exception: jsonschema 3.2.0 has no such extra feature 'format-nongpl')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/bin/jb", line 8, in <module>
    sys.exit(main())
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/jupyter_book/cli/main.py", line 323, in build
    builder_specific_actions(
  File "/Users/richardevans/opt/anaconda3/envs/oguk-dev/lib/python3.9/site-packages/jupyter_book/cli/main.py", line 531, in builder_specific_actions
    raise RuntimeError(_message_box(msg, color="red", doprint=False)) from result
RuntimeError: 
===============================================================================

There was an error in building your book. Look above for the cause.

===============================================================================

nikhilwoodruff commented 1 year ago

Thanks @rickecon - I just took a look at this PR locally and fixed some of the failing tests (I couldn't reproduce your one though), will see what the actions say.

nikhilwoodruff commented 1 year ago

OK @rickecon - think this is getting somewhere: the package installs are now working, and the microdata tests passing, but the run_oguk module is hitting an error, and I don't really understand it but it seems plausibly logic-related:

Warning: There is a discrepency between the start year of the model and that of the tax functions!!p.BW = 6, BW_in_tax_params = 80
Warning: There is a discrepency between the ages used in the model and those in the tax functions!!p.S = 80, S_in_tax_params = 6

So I think the BW in p is correct, right? Is the tax function estimation putting things in the wrong place? Thought that was unlikely because it's a call to OG-Core, but yeah any ideas?

Also, here's the shape of each item in the tax function parameter dictionary:

{
  'tfunc_etr_params_S': (6, 80, 1), 
  'tfunc_mtrx_params_S': (6, 80, 1), 
  'tfunc_mtry_params_S': (6, 80, 1), 
  'tfunc_avginc': (6,), 
  'tfunc_avg_etr': (6,), 
  'tfunc_avg_mtrx': (6,),
  'tfunc_avg_mtry': (6,), 
  'tfunc_frac_tax_payroll': (6,), 
  'tfunc_etr_sumsq': (6, 80), 
  'tfunc_mtrx_sumsq': (6, 80), 
  'tfunc_mtry_sumsq': (6, 80), 
  'tfunc_etr_obs': (6, 80),
  'tfunc_mtrx_obs': (6, 80), 
  'tfunc_mtry_obs': (6, 80), 
  'tfunc_time': (), 
  'tax_func_type': (), 
  'start_year': (), '
  BW': ()
}

codecov[bot] commented 1 year ago

Codecov Report

Base: 70.06% // Head: 72.91% // Increases project coverage by +2.84% :tada:

Coverage data is based on head (8b72361) compared to base (a72a51e). Patch coverage: 59.09% of modified lines in pull request are covered.

:exclamation: Current head 8b72361 differs from pull request most recent head a2d574c. Consider uploading reports for the commit a2d574c to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #57 +/- ## ========================================== + Coverage 70.06% 72.91% +2.84% ========================================== Files 4 4 Lines 147 144 -3 ========================================== + Hits 103 105 +2 + Misses 44 39 -5 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `72.91% <59.09%> (+2.84%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/PSLmodels/OG-UK/pull/57?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | Coverage Δ | | |---|---|---| | [setup.py](https://codecov.io/gh/PSLmodels/OG-UK/pull/57?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | | | [oguk/get\_micro\_data.py](https://codecov.io/gh/PSLmodels/OG-UK/pull/57?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1ay9nZXRfbWljcm9fZGF0YS5weQ==) | `72.46% <70.58%> (+0.40%)` | :arrow_up: | | [oguk/tests/test\_get\_micro\_data.py](https://codecov.io/gh/PSLmodels/OG-UK/pull/57?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2d1ay90ZXN0cy90ZXN0X2dldF9taWNyb19kYXRhLnB5) | `87.80% <100.00%> (+0.30%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

nikhilwoodruff commented 1 year ago

Just a few thoughts before our next OG call:

The steady-state is now consistently solving under linear tax functions (see non-last action test passes)
The steady-state is now consistently solving for me under DEP tax functions (but seems to be failing on Windows- see the last action)
All tests apart from the Windows test and the documentation build test are passing
The docs build test is failing because of a dependency error I've seen elsewhere (I think it's an issue with JupyterBook, but the fix I used for other repos, force-installing jsonschema[format-nongpl], didn't work here for some reason)

I'd recommend merging this PR without the Windows and docs build test passes because neither of them represent a regression really (the docs just won't get updated rather than being taken down), and also because I think this PR overlaps/aids some of the others (e.g. I think it fixes #55, and resolves pretty much all PolicyEngine-UK-related issues in other PRs).

rickecon commented 1 year ago

@nikhilwoodruff. I discovered that one reason my earlier runs of the model were breaking in the get_micro_data.py stage (see this output in the comment above) is that PolicyEngine-UK doesn't work with the Anaconda Python distribution for Mac OS (64-bit M1 optimized distribution). We found that this distribution sped up computation times by 33% (increase in computation going from standard Mac OS Python distribution to M1 optimized is 50%). For the time being, I have reverted my Python distribution to the standard Anaconda distribution.

However, I now get the following error when running on my machine in the oguk-dev conda environment which says that ReformType is not a defined type in line 45 of get_micro_data.py.

(oguk-dev) richardevans@Richards-MacBook-Pro OG-UK % python ./examples/run_oguk.py
WARNING:root:
    Could not locate FRS microdata. If you have access to the data, try running:

    policyengine-uk-data enhanced_frs download 2022

WARNING:root:Using synthetic FRS microdata.
INFO:root:Downloading 2022 synthetic FRS microdata.
100%|█████████████████████████████████████████████████████████████████| 27.0M/27.0M [00:03<00:00, 8.58MiB/s]
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-UK/./examples/run_oguk.py", line 9, in <module>
    from oguk.calibrate import Calibration
  File "/Users/richardevans/Docs/Economics/OSE/OG-UK/oguk/calibrate.py", line 2, in <module>
    from oguk import get_micro_data
  File "/Users/richardevans/Docs/Economics/OSE/OG-UK/oguk/get_micro_data.py", line 45, in <module>
    reform: ReformType,
NameError: name 'ReformType' is not defined

nikhilwoodruff commented 1 year ago

@rickecon - I had no idea about the M1-optimised Python distribution, interesting! (I think we might be on the same Mac device?).

Hmm, I thought I'd removed the ReformType reference, it's no longer in the repo for me?

PSLmodels / OG-UK

Update demographics.py and PolicyEngine #57

Codecov Report