search

piberger / Xbb

New repository for VHbb plus other Xbb analyses
0 stars 8 forks source link

Fixed memory leak in KinFitter #128

Closed dabercro closed 4 years ago

dabercro commented 4 years ago

The heap generated by running the kinematic fit had lots of "cons_x" and "cons_y" when run through strings, so I think the leak was actually from the creation of the constraints here. ROOT likes to store things in a global hash table, so Python's garbage collection doesn't always work.

I'm running jobs with all the systematics right now. They don't seem to cause any swapping yet after almost 2 hours, but I'll keep an eye on them.

piberger commented 4 years ago

I'll try that, thanks! I'm a bit surprised, since when I quickly looked at the memory leak, it seemed it is only present, when this call here is done: https://github.com/piberger/Xbb/blob/8238c22249bcf79e4a588ee8cbe86cf90e6d738e/python/myutils/kinFitterXbb.py#L283 commenting the above line removed the leak, even though the constraints still would be created. Maybe the fit() call creates additional strings not accessible by the Python gc.

dabercro commented 4 years ago

Ah okay, I think you're right. I think these lines helped my test job, but now running over the whole cluster, I'm still getting a lot of memory usage from TTTo2L jobs. I'll have to switch tests.

piberger commented 4 years ago

from my experience the leak is ~GBs for O(millions) of calls to fit()

dabercro commented 4 years ago

Yeah, the jobs were getting up to 2-3 GB each so I'll kill them for now and try again. Each of our machines has about 16 GB of memory, but they each run 8 jobs at the same time.

dabercro commented 4 years ago

I'll keep you updated if I find anything else. I think the strings in memory should help track it down though.