search

nanoomlee / HYREC-2

A code for cosmological recombination history (arXiv:2007.14114)
5 stars 5 forks source link

Memory leak #2

Closed carronj closed 3 years ago

carronj commented 3 years ago

In all my attempts to use HYREC-2 as recombination code for CAMB in a cobaya MCMC chains, memory accumulates until exhausted. Is memory properly deallocated in this code?

gcanasherrera commented 3 years ago

I have recently encountered the same issue (CAMB + HYREC + COBAYA). I have discarded Cobaya to be the responsible and it seems it is something inherent in HyRec.

gcanasherrera commented 3 years ago

Is there any update about this memory leak issue?

nanoomlee commented 3 years ago

Sorry for my late response and thank you for addressing the issue!

Can you please try with two simple modifications? In history.c,

  1. change line 94: "firstTime =1;" -> "firstTime =0;"
  2. add a line: "hyrec_free(&rec_data);" at the end of rec_build_historycamb() function.

I believe this should fix the issue. That "firstTime" flag was introduced to load data (e.g fitting function) only once during CAMB run. And this data is not freed since if it's loaded only once, this shouldn't be a problem.

And I thought this would work for MCMC too but it seems not (probably, the data is loaded multiple times without being freed in MCMC). So, if you make those modifications above, each time HYREC is called in CAMB, data will be loaded and freed at the end. It's kind of inefficient since the code will load the same data several times in one CAMB run (and also when python version of CAMB is used), but given that the time taking for loading data is very short, this will be fine for MCMC.

Please let me know if the modification above works! Thank you!

JesusTorrado commented 3 years ago

Hi @nanoomlee

Loading at exec should be OK in most MCMC setups, but ideally this would not be the definitive solution, since when running on clusters some partitions have faster I/O access time than others, and users are not always careful about that.

nanoomlee commented 3 years ago

Hi @JesusTorrado,

I see. Thanks for comments!

Anyway, @carronj confirmed the modification above doesn't work and I'm trying different modification but still by loading at exec.

But, can you guess at which part the memory leak happens? When I run Cobaya with CAMB and HYREC-2 (before any modification) using yaml file, I checked that the structure HYREC_DATA which stores all the important data (or results) is loaded only once for multiple CAMB run, so I think this shouldn't cause memory leak.

Also, when I run Cobaya + CAMB + HYREC-2, I actually get following error.

Thanks!

nanoomlee commented 3 years ago

@JesusTorrado @carronj @gcanasherrera

The memory leak was due to an unnecessary allocation in hyrecxe() function in history.c which is used in CAMB. Now code is fixed and seems not cause any memory leak anymore.

Thank you for addressing issue and for comments! In particular, many thanks to @carronj for helping me regarding this issue!