search

PSLmodels / Tax-Brain

Tax-Brain is an integrator model for PSL tax models
http://taxbrain.pslmodels.org/
MIT License
9 stars 14 forks source link

Idea for CS Speed Improvement - Behavioral Responses Package #98

Closed Peter-Metz closed 4 years ago

Peter-Metz commented 4 years ago

I have a hunch that part of the reason Tax-Brain takes a long time to run on C/S is because of the way the behavior parameters are handled. Right now, the user mods are processed with Paramtools using behavior_params.json and the default dictionary is:

defaults = {"sub": 0, "inc": 0, "cg": 0}

Which, whether or not the user modifies these parameters, is fed into the Tax-Brain API (can someone verify this?). In a Jupyter Notebook, I found that the following takes about 40 seconds to run:

tb = TaxBrain(2019, 2028, use_cps=True, reform=reform_url)
tb.run()

While this takes at least 3 minutes to run:

tb = TaxBrain(2019, 2028, use_cps=True, reform=reform_url, behavior={"sub": 0, "inc": 0, "cg": 0})
tb.run()

If my understanding of the behavior parameter handling is correct, my suggestion would be to avoid passing a behavioral responses dictionary into the Tax-Brain API if the user does not modify behavioral parameters. Happy to open a PR if others agree with this approach.

cc: @hdoupe @andersonfrailey @MattHJensen

hdoupe commented 4 years ago

@Peter-Metz I just looked into this and it looks like an empty dictionary is passed to Tax-Brain when behavior parameters are not modified. To test this, I commented out the behavior parameter "inc" in the cs-config tests, set verbose to True, and added a print statement for the behavior params dictionary: https://github.com/PSLmodels/Tax-Brain/compare/master...hdoupe:check-behavior-params

tb.run() finishes in in about 25 secs when an empty behavior mod dictionary is passed to run_model and 84 secs when non-empty behavior mod dictionary is passed to run_model. You can also see the print output of which method is called where it says "Running static simulations" and "Running dynamic simulations."

Also, on Compute Studio, the simulations where behavior parameters are modified take about 300-330 seconds while the static simulations take about 150 seconds.

With "inc" commented out:

(taxbrain-dev) hankdoupe@system76-pc:~/Documents/Tax-Brain$ py.test -v -s cs-config/cs_config/tests/test_functions.py::TestFunctions1::test_run_model
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.3, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- /home/hankdoupe/miniconda3/envs/taxbrain-dev/bin/python
cachedir: .pytest_cache
rootdir: /home/hankdoupe/Documents/Tax-Brain/cs-config
collected 1 item                                                                                                                                                                             

cs-config/cs_config/tests/test_functions.py::TestFunctions1::test_run_model Defaulting to the CPS
behavior_mods {}
You loaded data for 2014.
Tax-Calculator startup automatically extrapolated your data to 2014.
You loaded data for 2014.
Tax-Calculator startup automatically extrapolated your data to 2014.
Running static simulations
tb.run() took 23.331905603408813s
delaying for 2019
delaying for 2020
delaying for 2021
delaying for 2022
delaying for 2023
delaying for 2024
delaying for 2025
delaying for 2026
delaying for 2027
delaying for 2028
delaying for 2029
elapsed time for this run: 102.8
elapsed time for this run: 103.4
elapsed time for this run: 103.9
elapsed time for this run: 104.0
elapsed time for this run: 104.1
elapsed time for this run: 104.1
elapsed time for this run: 104.2
elapsed time for this run: 104.2
elapsed time for this run: 104.2
elapsed time for this run: 104.2
elapsed time for this run: 104.2
Write finished in 0.012156963348388672s
PASSED

====================================================================================== warnings summary ======================================================================================
/home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/numba/types/containers.py:3
  /home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/numba/types/containers.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

/home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/html5lib/_trie/_base.py:3
  /home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

cs_config/tests/test_functions.py::TestFunctions1::test_run_model
cs_config/tests/test_functions.py::TestFunctions1::test_run_model
  BokehDeprecationWarning: CoffeeScript support is deprecated and will be removed in an eventual 2.0 release. Use JavaScript or TypeScript directly instead.

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================================================================= 1 passed, 4 warnings in 139.90s (0:02:19) ==========================================================================

with "inc":

(taxbrain-dev) hankdoupe@system76-pc:~/Documents/Tax-Brain$ py.test -v -s cs-config/cs_config/tests/test_functions.py::TestFunctions1::test_run_model
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.3, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- /home/hankdoupe/miniconda3/envs/taxbrain-dev/bin/python
cachedir: .pytest_cache
rootdir: /home/hankdoupe/Documents/Tax-Brain/cs-config
collected 1 item                                                                                                                                                                             

cs-config/cs_config/tests/test_functions.py::TestFunctions1::test_run_model Defaulting to the CPS
behavior_mods {'inc': -0.1}
You loaded data for 2014.
Tax-Calculator startup automatically extrapolated your data to 2014.
You loaded data for 2014.
Tax-Calculator startup automatically extrapolated your data to 2014.
Running dynamic simulations
tb.run() took 84.92324733734131s
delaying for 2019
delaying for 2020
delaying for 2021
delaying for 2022
delaying for 2023
delaying for 2024
delaying for 2025
delaying for 2026
delaying for 2027
delaying for 2028
delaying for 2029
elapsed time for this run: 103.5
elapsed time for this run: 103.6
elapsed time for this run: 103.7
elapsed time for this run: 103.8
elapsed time for this run: 104.0
elapsed time for this run: 104.0
elapsed time for this run: 104.1
elapsed time for this run: 104.1
elapsed time for this run: 104.2
elapsed time for this run: 104.2
elapsed time for this run: 104.2
Write finished in 0.011918306350708008s
PASSED

====================================================================================== warnings summary ======================================================================================
/home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/numba/types/containers.py:3
  /home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/numba/types/containers.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

/home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/html5lib/_trie/_base.py:3
  /home/hankdoupe/miniconda3/envs/taxbrain-dev/lib/python3.7/site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

cs_config/tests/test_functions.py::TestFunctions1::test_run_model
cs_config/tests/test_functions.py::TestFunctions1::test_run_model
  BokehDeprecationWarning: CoffeeScript support is deprecated and will be removed in an eventual 2.0 release. Use JavaScript or TypeScript directly instead.

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================================================================= 1 passed, 4 warnings in 201.37s (0:03:21) ==========================================================================
Peter-Metz commented 4 years ago

Thanks for looking into this @hdoupe! I'll close this issue because it doesn't seem to be slowing things down.

hdoupe commented 4 years ago

No problem! Thanks for bringing this up. Checking simple stuff like this is good to do before we start doing more complicated things to improve performance. I'm also excited people are thinking about ways to speed things up in the taxcalc/taxbrain ecosystem.