search

21cmfast / 21cmFAST

Official repository for 21cmFAST: a code for generating fast simulations of the cosmological 21cm signal
MIT License
58 stars 38 forks source link

Improve memory usage #224

Closed BradGreig closed 3 years ago

BradGreig commented 3 years ago

This PR aims to minimise memory usage. It takes in the flag that has been introduced in #220 (MINIMIZE_MEMORY).

It improves memory usage in two places, only one of which uses this new flag. The first is in generating the ICs, where instead of creating 6 new boxes for the components of the 2LPT, I instead temporarily store the i=j components in the hires_vi_2LPT boxes which have been allocated (but not populated) at this point (they get filled after) and only need a single box for the current i>j component. This results in 5 less boxes (of size DIM). As this minimises memory without additional CPU/IO this change is to always be implemented.

The one usage of MINIMIZE_MEMORY so far is in SpinTemperatureBox.c. It is only used when USE_MASS_DEPENDENT_ZETA=True and USE_INTERPOLATION_TABLES=True. It results in only allocating one box for the current smoothed density at the expense of global_params.NUM_FILTER_STEPS_FOR_Ts additional FFT's per redshift (but having global_params.NUM_FILTER_STEPS_FOR_Ts - 1 less boxes in memory). This doesn't seem to add too much in terms of overall computations, especially if USE_FFTW_WISDOM=True.

Closes #63

codecov[bot] commented 3 years ago

Codecov Report

Merging #224 (164d8f6) into master (0bdd060) will not change coverage. The diff coverage is n/a.

:exclamation: Current head 164d8f6 differs from pull request most recent head 1c77d0d. Consider uploading reports for the commit 1c77d0d to get more accurate results Impacted file tree graph

@@           Coverage Diff           @@
##           master     #224   +/-   ##
=======================================
  Coverage   84.81%   84.81%           
=======================================
  Files          12       12           
  Lines        2444     2444           
=======================================
  Hits         2073     2073           
  Misses        371      371
Impacted Files Coverage Δ
src/py21cmfast/inputs.py 90.60% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 45973ad...1c77d0d. Read the comment docs.

steven-murray commented 3 years ago

I'd approve if it was ready for it :-)

BradGreig commented 3 years ago

Yeah, I'll only move it to a full PR once #220 gets pulled into master and I merge all that in.

BradGreig commented 3 years ago

Hey @steven-murray, I added the test data and fixed up the original linting issue, however, it is still failing the linting. I tried re-running the linting tests, which gave the existing error. The previous error was a timing out issue. Might have been an issue on their server end or something...