search

PSLmodels / OG-Core

An overlapping generations model framework for evaluating fiscal policies.
https://pslmodels.github.io/OG-Core/
Creative Commons Zero v1.0 Universal
68 stars 119 forks source link

Remove `surv_rate` parameter object #886

Closed jdebacker closed 1 year ago

jdebacker commented 1 year ago

This PR removes the surv_rate parameter object from OG-Core. This object is redundant with the mortality rates (rho).

surv_rate is removed from the following files:

Partially addresses Issue #885

codecov-commenter commented 1 year ago

Codecov Report

Merging #886 (75ec1cf) into master (0dfe9c7) will decrease coverage by 0.02%. The diff coverage is 66.66%.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886/graphs/tree.svg?width=650&height=150&src=pr&token=98mQCVhspd&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels)](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) ```diff @@ Coverage Diff @@ ## master #886 +/- ## ========================================== - Coverage 79.83% 79.82% -0.02% ========================================== Files 18 18 Lines 4181 4183 +2 ========================================== + Hits 3338 3339 +1 - Misses 843 844 +1 ``` | [Flag](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | `79.82% <66.66%> (-0.02%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | Coverage Δ | | |---|---|---| | [ogcore/\_\_init\_\_.py](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2djb3JlL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [ogcore/output\_tables.py](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2djb3JlL291dHB1dF90YWJsZXMucHk=) | `96.75% <100.00%> (+0.01%)` | :arrow_up: | | [ogcore/parameters.py](https://app.codecov.io/gh/PSLmodels/OG-Core/pull/886?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#diff-b2djb3JlL3BhcmFtZXRlcnMucHk=) | `72.31% <0.00%> (-0.31%)` | :arrow_down: |
rickecon commented 1 year ago

@jdebacker. I just added a PR to your branch that updates the version number in setup.py and in ogcore/__init__.py. This will allow an update to the ogcore PyPI package, which will allow our country calibration packages to have the right dependencies.

rickecon commented 1 year ago

@jdebacker. I am running into an error when I run pytest on my local machine. I have rebuilt the ogcore-dev conda environment. And when I type generic pytest in my terminal, I get the following error. In each case, it is an unpickling error, which makes me think that removing surv_rate will require us to update all the testing pickles. I was getting this error yesterday when I was trying to add this removal of surv_rate to @harisjoshi 's PR. But why are none of our GH Action CI tests failing? Let me know if you can recreate this.

(ogcore-dev) richardevans@Richards-MacBook-Pro-2 OG-Core % pytest
=================================== test session starts ===================================
platform darwin -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: /Users/richardevans/Docs/Economics/OSE/OG-Core
configfile: pytest.ini
testpaths: ./tests
plugins: xdist-3.3.1
collected 367 items / 5 errors                                                            

========================================= ERRORS ==========================================
___________________________ ERROR collecting tests/test_TPI.py ____________________________
ogcore/utils.py:393: in safe_read_pickle
    obj = pickle.load(f, encoding="latin1")
E   TypeError: code() argument 13 must be str, not int

During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_TPI.py:573: in <module>
    dict_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
    obj = pickle.load(f)  # pragma no cover
E   _pickle.UnpicklingError: Memo value not found at index 80
_______________________ ERROR collecting tests/test_output_plots.py _______________________
ogcore/utils.py:393: in safe_read_pickle
    obj = pickle.load(f, encoding="latin1")
E   TypeError: code() argument 13 must be str, not int

During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_output_plots.py:20: in <module>
    base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
    obj = pickle.load(f)  # pragma no cover
E   _pickle.UnpicklingError: Memo value not found at index 15776
______________________ ERROR collecting tests/test_output_tables.py _______________________
ogcore/utils.py:393: in safe_read_pickle
    obj = pickle.load(f, encoding="latin1")
E   TypeError: code() argument 13 must be str, not int

During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_output_tables.py:20: in <module>
    base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
    obj = pickle.load(f)  # pragma no cover
E   _pickle.UnpicklingError: Memo value not found at index 15776
_____________________ ERROR collecting tests/test_parameter_plots.py ______________________
ogcore/utils.py:393: in safe_read_pickle
    obj = pickle.load(f, encoding="latin1")
E   TypeError: code() argument 13 must be str, not int

During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_parameter_plots.py:16: in <module>
    base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
    obj = pickle.load(f)  # pragma no cover
E   _pickle.UnpicklingError: Memo value not found at index 15776
_____________________ ERROR collecting tests/test_parameter_tables.py _____________________
ogcore/utils.py:393: in safe_read_pickle
    obj = pickle.load(f, encoding="latin1")
E   TypeError: code() argument 13 must be str, not int

During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_parameter_tables.py:14: in <module>
    base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
    obj = pickle.load(f)  # pragma no cover
E   _pickle.UnpicklingError: Memo value not found at index 15776
==================================== warnings summary =====================================
ogcore/utils.py:12
  /Users/richardevans/Docs/Economics/OSE/OG-Core/ogcore/utils.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import resource_stream, Requirement

../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944
  /opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944: RemovedInMarshmallow4Warning: The 'missing' argument to fields is deprecated. Use 'load_default' instead.
    super().__init__(**kwargs)

../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:569
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:569
  /opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:569: RemovedInMarshmallow4Warning: The 'missing' argument to fields is deprecated. Use 'load_default' instead.
    super().__init__(default=default, dump_default=dump_default, **kwargs)

../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:1558
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:1558
  /opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:1558: RemovedInMarshmallow4Warning: The 'missing' argument to fields is deprecated. Use 'load_default' instead.
    super().__init__(**kwargs)

tests/test_aggregates.py:31: 12800 warnings
  /Users/richardevans/Docs/Economics/OSE/OG-Core/tests/test_aggregates.py:31: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    L_loop[t, i, k] *= (

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================= short test summary info =================================
ERROR tests/test_TPI.py - _pickle.UnpicklingError: Memo value not found at index 80
ERROR tests/test_output_plots.py - _pickle.UnpicklingError: Memo value not found at index 15776
ERROR tests/test_output_tables.py - _pickle.UnpicklingError: Memo value not found at index 15776
ERROR tests/test_parameter_plots.py - _pickle.UnpicklingError: Memo value not found at index 15776
ERROR tests/test_parameter_tables.py - _pickle.UnpicklingError: Memo value not found at index 15776
!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!
=========================== 12808 warnings, 5 errors in 49.55s ============================
jdebacker commented 1 year ago

@rickecon I think this is a Python 3.11 thing. I posted a similar issue in the UN-OG-Training repo today.

Will need to see if we can find a way to read these pickle files created in an earlier Python or if we need to recreate them all.

rickecon commented 1 year ago

@jdebacker For now, maybe we should just pin to Python 3.10 (python>=3.10,<3.11) in environment.yml and setup.py, and leave the pickle adjustment to an issue and another PR. I don't see any reason to keep Python 3.9 as long as Tax-Calculator and FiscalSim and PolicyEngine using Python 3.10 (which they all do as of last weekend). This would also cut our GH Actions CI time in half to get rid of the Python 3.9 testing. But we could also just pin to 3.9 or 3.10 for now (python>=3.9,<3.11).

jdebacker commented 1 year ago

@rickecon I wanted to look around at a solution to the unpickling issue since I would like to keep as loose of Python version pins as possible (being flexible here is important, for example, in some government bureaucracies where it takes many months to get new software versions approved).

The issue affecting us here (not being able to read pickle files created in older versions of Python) only affects files created with cloudpickle as that protocol does not allow for compatibility between different version of Python (e.g., see the discussion in this issue). Fortunately, we only have (I believe) four files saved with cloudpickle cached in this repo (all in OG-Core/tests/test_io_data/):

  1. run_TPI_outputs_mono_2.pkl
  2. TxFuncEst_mono_nonage.pkl
  3. model_params_baseline.pkl
  4. model_params_reform.pkl

Let me try to recreate versions of these saved with cloudpickle in Python 3.11 and set up some conditional statements to select the right file when testing. I'll make those changes in this PR.

jdebacker commented 1 year ago

@rickecon Tests are now passing, here's what I did with respect to the four files noted above:

  1. run_TPI_outputs_mono_2.pkl -> bypass the test case of test_run_TPI_extra where mono tax functions are used if the version is >= 3.11
  2. TxFuncEst_mono_nonage.pkl -> bypass the test case of test_plot_2D_taxfunc where mono tax functions are used if the version is >= 3.11
  3. model_params_baseline.pkl -> added model_params_baseline_v311.pkl that is used when the Python version is >= 3.11
  4. model_params_reform.pkl -> added model_params_reform_v311.pkl that is used when the Python version is >= 3.11

Note, the reason I chose to bypass tests for cases (1) and (2) is because, at the moment, it's been difficult to generate new mono tax function estimates in OG-USA (see the notes in PR #73).

jdebacker commented 1 year ago

@rickecon Thanks for your review. I'm going to merge this. I'm ok with the time on GH Actions (things run in parallel anyway) and really like ensuring compatibility with older versions of Python for the reasons expressed above.