Closed jdebacker closed 1 year ago
Merging #886 (75ec1cf) into master (0dfe9c7) will decrease coverage by
0.02%
. The diff coverage is66.66%
.
@jdebacker. I just added a PR to your branch that updates the version number in setup.py
and in ogcore/__init__.py
. This will allow an update to the ogcore
PyPI package, which will allow our country calibration packages to have the right dependencies.
@jdebacker. I am running into an error when I run pytest
on my local machine. I have rebuilt the ogcore-dev
conda environment. And when I type generic pytest
in my terminal, I get the following error. In each case, it is an unpickling error, which makes me think that removing surv_rate
will require us to update all the testing pickles. I was getting this error yesterday when I was trying to add this removal of surv_rate
to @harisjoshi 's PR. But why are none of our GH Action CI tests failing? Let me know if you can recreate this.
(ogcore-dev) richardevans@Richards-MacBook-Pro-2 OG-Core % pytest
=================================== test session starts ===================================
platform darwin -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: /Users/richardevans/Docs/Economics/OSE/OG-Core
configfile: pytest.ini
testpaths: ./tests
plugins: xdist-3.3.1
collected 367 items / 5 errors
========================================= ERRORS ==========================================
___________________________ ERROR collecting tests/test_TPI.py ____________________________
ogcore/utils.py:393: in safe_read_pickle
obj = pickle.load(f, encoding="latin1")
E TypeError: code() argument 13 must be str, not int
During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
???
<frozen importlib._bootstrap>:1176: in _find_and_load
???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:690: in _load_unlocked
???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
tests/test_TPI.py:573: in <module>
dict_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
obj = pickle.load(f) # pragma no cover
E _pickle.UnpicklingError: Memo value not found at index 80
_______________________ ERROR collecting tests/test_output_plots.py _______________________
ogcore/utils.py:393: in safe_read_pickle
obj = pickle.load(f, encoding="latin1")
E TypeError: code() argument 13 must be str, not int
During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
???
<frozen importlib._bootstrap>:1176: in _find_and_load
???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:690: in _load_unlocked
???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
tests/test_output_plots.py:20: in <module>
base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
obj = pickle.load(f) # pragma no cover
E _pickle.UnpicklingError: Memo value not found at index 15776
______________________ ERROR collecting tests/test_output_tables.py _______________________
ogcore/utils.py:393: in safe_read_pickle
obj = pickle.load(f, encoding="latin1")
E TypeError: code() argument 13 must be str, not int
During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
???
<frozen importlib._bootstrap>:1176: in _find_and_load
???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:690: in _load_unlocked
???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
tests/test_output_tables.py:20: in <module>
base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
obj = pickle.load(f) # pragma no cover
E _pickle.UnpicklingError: Memo value not found at index 15776
_____________________ ERROR collecting tests/test_parameter_plots.py ______________________
ogcore/utils.py:393: in safe_read_pickle
obj = pickle.load(f, encoding="latin1")
E TypeError: code() argument 13 must be str, not int
During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
???
<frozen importlib._bootstrap>:1176: in _find_and_load
???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:690: in _load_unlocked
???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
tests/test_parameter_plots.py:16: in <module>
base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
obj = pickle.load(f) # pragma no cover
E _pickle.UnpicklingError: Memo value not found at index 15776
_____________________ ERROR collecting tests/test_parameter_tables.py _____________________
ogcore/utils.py:393: in safe_read_pickle
obj = pickle.load(f, encoding="latin1")
E TypeError: code() argument 13 must be str, not int
During handling of the above exception, another exception occurred:
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:531: in collect
self._inject_setup_module_fixture()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:310: in obj
self._obj = obj = self._getobj()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
return self._importtestmodule()
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
???
<frozen importlib._bootstrap>:1176: in _find_and_load
???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:690: in _load_unlocked
???
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
tests/test_parameter_tables.py:14: in <module>
base_params = utils.safe_read_pickle(
ogcore/utils.py:395: in safe_read_pickle
obj = pickle.load(f) # pragma no cover
E _pickle.UnpicklingError: Memo value not found at index 15776
==================================== warnings summary =====================================
ogcore/utils.py:12
/Users/richardevans/Docs/Economics/OSE/OG-Core/ogcore/utils.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import resource_stream, Requirement
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:944: RemovedInMarshmallow4Warning: The 'missing' argument to fields is deprecated. Use 'load_default' instead.
super().__init__(**kwargs)
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:569
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:569
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:569: RemovedInMarshmallow4Warning: The 'missing' argument to fields is deprecated. Use 'load_default' instead.
super().__init__(default=default, dump_default=dump_default, **kwargs)
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:1558
../../../../../../opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:1558
/opt/anaconda3/envs/ogcore-dev/lib/python3.11/site-packages/marshmallow/fields.py:1558: RemovedInMarshmallow4Warning: The 'missing' argument to fields is deprecated. Use 'load_default' instead.
super().__init__(**kwargs)
tests/test_aggregates.py:31: 12800 warnings
/Users/richardevans/Docs/Economics/OSE/OG-Core/tests/test_aggregates.py:31: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
L_loop[t, i, k] *= (
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================= short test summary info =================================
ERROR tests/test_TPI.py - _pickle.UnpicklingError: Memo value not found at index 80
ERROR tests/test_output_plots.py - _pickle.UnpicklingError: Memo value not found at index 15776
ERROR tests/test_output_tables.py - _pickle.UnpicklingError: Memo value not found at index 15776
ERROR tests/test_parameter_plots.py - _pickle.UnpicklingError: Memo value not found at index 15776
ERROR tests/test_parameter_tables.py - _pickle.UnpicklingError: Memo value not found at index 15776
!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!
=========================== 12808 warnings, 5 errors in 49.55s ============================
@rickecon I think this is a Python 3.11 thing. I posted a similar issue in the UN-OG-Training repo today.
Will need to see if we can find a way to read these pickle files created in an earlier Python or if we need to recreate them all.
@jdebacker For now, maybe we should just pin to Python 3.10 (python>=3.10,<3.11
) in environment.yml
and setup.py
, and leave the pickle adjustment to an issue and another PR. I don't see any reason to keep Python 3.9 as long as Tax-Calculator and FiscalSim and PolicyEngine using Python 3.10 (which they all do as of last weekend). This would also cut our GH Actions CI time in half to get rid of the Python 3.9 testing. But we could also just pin to 3.9 or 3.10 for now (python>=3.9,<3.11
).
@rickecon I wanted to look around at a solution to the unpickling issue since I would like to keep as loose of Python version pins as possible (being flexible here is important, for example, in some government bureaucracies where it takes many months to get new software versions approved).
The issue affecting us here (not being able to read pickle files created in older versions of Python) only affects files created with cloudpickle
as that protocol does not allow for compatibility between different version of Python (e.g., see the discussion in this issue). Fortunately, we only have (I believe) four files saved with cloudpickle
cached in this repo (all in OG-Core/tests/test_io_data/
):
run_TPI_outputs_mono_2.pkl
TxFuncEst_mono_nonage.pkl
model_params_baseline.pkl
model_params_reform.pkl
Let me try to recreate versions of these saved with cloudpickle
in Python 3.11 and set up some conditional statements to select the right file when testing. I'll make those changes in this PR.
@rickecon Tests are now passing, here's what I did with respect to the four files noted above:
run_TPI_outputs_mono_2.pkl
-> bypass the test case of test_run_TPI_extra
where mono tax functions are used if the version is >= 3.11TxFuncEst_mono_nonage.pkl
-> bypass the test case of test_plot_2D_taxfunc
where mono tax functions are used if the version is >= 3.11model_params_baseline.pkl
-> added model_params_baseline_v311.pkl
that is used when the Python version is >= 3.11model_params_reform.pkl
-> added model_params_reform_v311.pkl
that is used when the Python version is >= 3.11Note, the reason I chose to bypass tests for cases (1) and (2) is because, at the moment, it's been difficult to generate new mono tax function estimates in OG-USA (see the notes in PR #73).
@rickecon Thanks for your review. I'm going to merge this. I'm ok with the time on GH Actions (things run in parallel anyway) and really like ensuring compatibility with older versions of Python for the reasons expressed above.
This PR removes the
surv_rate
parameter object from OG-Core. This object is redundant with the mortality rates (rho
).surv_rate
is removed from the following files:default_parameters.json
parameters.py
, wheresurv_rate
is used to calculate the SS population distribution in the case of constant demographics. Here, we replacesurv_rate
with1 - rho
.testing_parameters.json
Partially addresses Issue #885