Write code to explicitly clean up python objects that may be taking up memory at the end of a single task in the pipeline.

BSDExabio / PSP

protein structure prediction

1 stars 0 forks source link

Write code to explicitly clean up python objects that may be taking up memory at the end of a single task in the pipeline. #18

Closed rbdavid closed 2 years ago

rbdavid commented 2 years ago

A run of the minimization dask pipeline fails for a job with 25k tasks, while other jobs with fewer tasks survive just fine. Mark brought up a good point about objects in the task python script not being cleaned up after a single iteration. Will be adding code to remove python variables/objects from memory, leaving them to be cleaned up during garbage collection.

rbdavid commented 2 years ago

Implemented a variety of del lines to remove references to a specific variables/objects as a means to indicate garbage collection to clean up memory. Nothing really changed after this implementation though.