search

PSLmodels / OG-USA

Overlapping-generations macroeconomic model for evaluating fiscal policy in the United States
https://pslmodels.github.io/OG-USA/
Creative Commons Zero v1.0 Universal
19 stars 34 forks source link

Dealing with Dask memory issues in the run script, #2 #87

Closed jdebacker closed 5 months ago

jdebacker commented 5 months ago

Following up on PR #85, I noticed that despite this changes I was again seeing memory warnings:

2024-02-08 09:49:00,004 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)

And no, or slow, progress in when running run_og_usa.py when running with OG-Core 0.11.0 (which increases the size of several objects in the Specifications class object that contains the model parameters.

After some profiling, I noticed that the "bytes stored" in the dask processes became quite large with the tax function estimation, but then remained quite high after that task completed:

Dask_status_ogusa_taxfunc_big_memory

This PR helps deal with that issue by closing and deleting the client after tax function estimation and then instantiating a new client before the model solution algorithm is called.

This has seemed to avoid these issues.

rickecon commented 5 months ago

@jdebacker. This looks great. Thanks for doing this. Will merge as soon as tests pass.

codecov-commenter commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (7a6757b) 70.82% compared to head (b726392) 70.82%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #87 +/- ## ======================================= Coverage 70.82% 70.82% ======================================= Files 22 22 Lines 1340 1340 ======================================= Hits 949 949 Misses 391 391 ``` | [Flag](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/87/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/PSLmodels/OG-USA/pull/87/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels) | `70.82% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PSLmodels#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

rickecon commented 5 months ago

Thanks @jdebacker. Merging.