uclchem / UCLCHEM

UCLCHEM - A gas-grain chemical code for Astrochemistry.
MIT License
29 stars 15 forks source link

Increasing memory footprint #38

Closed GijsVermarien closed 1 year ago

GijsVermarien commented 1 year ago

Current version of UCLCHEM footprint increases when the python wrapper is used in a script to perform multiple models in a row, which is the default scenario for our grids: https://uclchem.github.io/docs/running_a_grid.

It was determined that this was probably due to the interplay of Fortran and f2py. Since we cannot really address it directly, the best solution is to make sure workers are restarted every so often. This can be done using the joblib workers, i.e. with Parallel and delayed:

from joblib import Parallel, delayed
results = Parallel(n_jobs=6)(
        delayed(pool_func)(row) for row in model_table.iterrows()
    )
GijsVermarien commented 1 year ago

Turns out this is too complicated to fix. Using joblib for grids and restarting workers can circumvent the issue.