Closed dabercro closed 4 years ago
I'll try that, thanks! I'm a bit surprised, since when I quickly looked at the memory leak, it seemed it is only present, when this call here is done: https://github.com/piberger/Xbb/blob/8238c22249bcf79e4a588ee8cbe86cf90e6d738e/python/myutils/kinFitterXbb.py#L283 commenting the above line removed the leak, even though the constraints still would be created. Maybe the fit() call creates additional strings not accessible by the Python gc.
Ah okay, I think you're right. I think these lines helped my test job, but now running over the whole cluster, I'm still getting a lot of memory usage from TTTo2L jobs. I'll have to switch tests.
from my experience the leak is ~GBs for O(millions) of calls to fit()
Yeah, the jobs were getting up to 2-3 GB each so I'll kill them for now and try again. Each of our machines has about 16 GB of memory, but they each run 8 jobs at the same time.
I'll keep you updated if I find anything else. I think the strings in memory should help track it down though.
The heap generated by running the kinematic fit had lots of "cons_x" and "cons_y" when run through
strings
, so I think the leak was actually from the creation of the constraints here. ROOT likes to store things in a global hash table, so Python's garbage collection doesn't always work.I'm running jobs with all the systematics right now. They don't seem to cause any swapping yet after almost 2 hours, but I'll keep an eye on them.