Closed martinholmer closed 7 months ago
jdebacker
commented
7 months ago @martinholmer Thanks for flagging this. It's coming from a file missing that should be there. This came from me reviewing PR #2709 too quickly and not catching the file deletion included in that PR.
jdebacker
commented
7 months ago @martinholmer I just merged PR #2712, so if you sync to master I do not think you will run into the issues noted here.
martinholmer
commented
7 months ago @jdebacker, Thanks for the quick response to issue #2711.
I don't get any errors now, but I do get a ton of warnings. See below for the different types of warnings.
First question: Why are we timing the tests and where do the benchmark test execution times come from?
Second question:
Why am I getting this pytest error? Does it have to do with the timing of the tests?
Traceback (most recent call last):
File "//Users/mrh/anaconda3/envs/taxcalc-dev/bin/pytest", line 10, in <module>
sys.exit(console_main())
^^^^^^^^^^^^^^Here is the complete (and highly confusing) results of "make pytest-cps":
(taxcalc-dev) Tax-Calculator% ./gitsync
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 13 (delta 9), reused 10 (delta 7), pack-reused 0
Unpacking objects: 100% (13/13), 11.49 KiB | 1.15 MiB/s, done.
From https://github.com/PSLmodels/Tax-Calculator
6528ca7b..552b2f88 master -> upstream/master
Updating 6528ca7b..552b2f88
Fast-forward
taxcalc/tests/test_stats_benchmark.csv | 335 +++++++++++++++++++++++++++++++++
taxcalc/utils.py | 4 +-
2 files changed, 337 insertions(+), 2 deletions(-)
create mode 100644 taxcalc/tests/test_stats_benchmark.csv
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/martinholmer/Tax-Calculator.git
6528ca7b..552b2f88 master -> master
(taxcalc-dev) Tax-Calculator% make pytest-cps
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
============================= test session starts ==============================
platform darwin -- Python 3.11.7, pytest-7.4.4, pluggy-1.0.0
rootdir: /Users/mrh/work/Tax-Calculator
configfile: pytest.ini
plugins: harvest-1.10.4, xdist-3.5.0, pep8-1.0.6
initialized: 4/4 workers//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
4 workers [278 items]
........................................................................ [ 25%]
........................................................................ [ 51%]
........................................................................ [ 77%]
.............................................................. [100%]taxcalc/tests/test_calculator.py::test_calculator_mtr is slower than the current benchmark by 7568.432 ms
taxcalc/tests/test_calculator.py::test_ID_HC_vs_BS is slower than the current benchmark by 1146.178 ms
taxcalc/tests/test_calculator.py::test_ID_StateLocal_HC_vs_CRT is slower than the current benchmark by 1678.292 ms
taxcalc/tests/test_calculator.py::test_ID_RealEstate_HC_vs_CRT is slower than the current benchmark by 1882.211 ms
taxcalc/tests/test_policy.py::TestAdjust::test_apply_cpi_offset is slower than the current benchmark by 1326.902 ms
taxcalc/tests/test_taxcalcio.py::test_output_options is slower than the current benchmark by 3920.533 ms
taxcalc/tests/test_taxcalcio.py::test_no_tables_or_graphs is slower than the current benchmark by 1019.528 ms
taxcalc/tests/test_calcfunctions.py::test_EITCamount[test_tuple0-6660] is slower than the current benchmark by 2047.391 ms
taxcalc/tests/test_calculator.py::test_qbid_limit_switch is slower than the current benchmark by 20668.816 ms
taxcalc/tests/test_calculator.py::test_calc_all_benefits_amounts is slower than the current benchmark by 1168.572 ms
taxcalc/tests/test_consumption.py::test_consumption_response is slower than the current benchmark by 1126.08 ms
taxcalc/tests/test_cpscsv.py::test_agg is slower than the current benchmark by 13421.672 ms
taxcalc/tests/test_policy.py::test_index_offset_reform is slower than the current benchmark by 1045.506 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2019] is slower than the current benchmark by 2254.137 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2020] is slower than the current benchmark by 1798.684 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2021] is slower than the current benchmark by 2165.293 ms
taxcalc/tests/test_taxcalcio.py::test_write_doc_file is slower than the current benchmark by 1906.345 ms
taxcalc/tests/test_calculator.py::test_reform_documentation is slower than the current benchmark by 1387.016 ms
taxcalc/tests/test_calculator.py::test_distribution_tables is slower than the current benchmark by 21282.246 ms
taxcalc/tests/test_calculator.py::test_ce_aftertax_income is slower than the current benchmark by 1155.073 ms
taxcalc/tests/test_policy.py::TestAdjust::test_multiple_cpi_swaps2 is slower than the current benchmark by 1060.249 ms
taxcalc/tests/test_policy.py::TestAdjust::test_adj_CPI_offset_and_index_status is slower than the current benchmark by 1213.617 ms
taxcalc/tests/test_policy.py::TestAdjust::test_cpi_offset_does_not_affect_wage_indexed_params is slower than the current benchmark by 1273.254 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2022] is slower than the current benchmark by 2263.939 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2023] is slower than the current benchmark by 1899.54 ms
taxcalc/tests/test_reforms.py::test_reform_json_and_output is slower than the current benchmark by 8708.887 ms
taxcalc/tests/test_benefits.py::test_benefits is slower than the current benchmark by 1901.282 ms
taxcalc/tests/test_parameters.py::test_json_file_contents[policy_current_law.json] is slower than the current benchmark by 15853.787 ms
taxcalc/tests/test_taxcalcio.py::test_creation_with_aging is slower than the current benchmark by 2442.26 ms
taxcalc/tests/test_taxcalcio.py::test_ctor_init_with_cps_files is slower than the current benchmark by 6147.77 ms
taxcalc/tests/test_taxcalcio.py::test_graphs is slower than the current benchmark by 20176.115 ms
=============================== warnings summary ===============================
Traceback (most recent call last):
File "//Users/mrh/anaconda3/envs/taxcalc-dev/bin/pytest", line 10, in <module>
sys.exit(console_main())
^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/main.py", line 306, in wrap_session
config.hook.pytest_sessionfinish(
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall
gen.send(outcome)
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 867, in pytest_sessionfinish
self.config.hook.pytest_terminal_summary(
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 34, in _multicall
next(gen) # first yield
^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 883, in pytest_terminal_summary
self.summary_warnings()
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 997, in summary_warnings
maybe_location = collapsed_location_report(message_reports)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 979, in collapsed_location_report
location = w.get_location(self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 337, in get_location
relpath = bestrelpath(config.invocation_params.dir, absolutepath(filename))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/pathlib.py", line 770, in bestrelpath
reldest = dest.relative_to(base)
^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/pathlib.py", line 730, in relative_to
raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/fsspec/registry.py' is not in the subpath of '/Users/mrh' OR one path is relative and the other is absolute.
make: *** [pytest-cps] Error 1
(taxcalc-dev) Tax-Calculator% >....
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 34, in _multicall
next(gen) # first yield
^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 883, in pytest_terminal_summary
self.summary_warnings()
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 997, in summary_warnings
maybe_location = collapsed_location_report(message_reports)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 979, in collapsed_location_report
location = w.get_location(self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 337, in get_location
relpath = bestrelpath(config.invocation_params.dir, absolutepath(filename))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/pathlib.py", line 770, in bestrelpath
reldest = dest.relative_to(base)
^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/pathlib.py", line 730, in relative_to
raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/fsspec/registry.py' is not in the subpath of '/Users/mrh' OR one path is relative and the other is absolute.
make: *** [pytest-cps] Error 1@martinholmer asks:
First question: Why are we timing the tests and where do the benchmark test execution times come from?
The performance benchmarking was introduced in PR #2570 to provide automated profiling of changes in performance for new PRs. At the time, work was being done to consider vectorized code with Dask arrays rather than Numba-jitted functions, so this was especially relevant. The test execution times are reported from pytest and gathered and formatted in Line 138 of conftest.py.
Second question: Why am I getting this pytest error? Does it have to do with the timing of the tests?
I do not know. It looks like all the tests in pytest passed for you. I don't know what to mark of the warnings summary. I actually have a much longer list of warnings, with notes about future deprecations in the Pandas API, etc.
martinholmer
commented
7 months ago @jdebacker said in issue #2711:
The performance benchmarking was introduced in PR https://github.com/PSLmodels/Tax-Calculator/pull/2570 to provide automated profiling of changes in performance for new PRs. At the time, work was being done to consider vectorized code with Dask arrays rather than Numba-jitted functions, so this was especially relevant.
OK. Is a change from numba to dask still under consideration? If not, can we somehow skip the pytest_sessionfinish because it keeps leaving a file on my disk that is not part of the repository. This is just harassment of developers whose only sin is to have a slower computer than the one used to establish the "benchmark" timings. After running "make pytest-cps", running "git status" produces this:
Untracked files:
(use "git add <file>..." to include in what will be committed)
taxcalc/tests/test_stats_current.csvWhat am I supposed to do with this untracked file?
I'm getting an error I don't understand when I do "make pytest-cps" on my computer using the code on the master branch. What am I doing wrong?