search

DUNE / dune-tms

DUNE ND Temporary Muon Spectrometer
0 stars 2 forks source link

Memory leak when running with overlay #23

Closed jdkio closed 1 year ago

jdkio commented 1 year ago

There's a memory leak somewhere that prevents the TMS reco from running with overlay. This was not an issue with the last sample run with overlay so it must have been introduced recently. That was run circa July but it's not clear if I was running with an older version of the reco.

For example, ConvertToTMSTree.exe /pnfs/dune/persistent/users/kleykamp/example_edep_file_with_overlay.root will crash because it runs out of memory.

clarenc3 commented 1 year ago

I can reproduce this with Jeffrey's file, but don't think it's a memory leak. The usual TMS reco seems to run fine, and when I manually do overlays it works fine to overlay e.g. 10 events.

It could simply be that the events are so large that it runs out of memory?

clarenc3 commented 1 year ago

During the meeting today we said that it likely is simply due to the event size; the time slicing should really happen before the reco stage. Will look into this

jdkio commented 1 year ago

Turns out it's not a memory leak nor due to event size. TMS_Event loops over all primary vertices and then within that loop gets all the hits, not just the subset associated with each vertex. So you end up with ~190x as many hits per event which runs out of memory before finishing the loop. Moved the code that loops through hits outside the primary vertices loop to fix.

jdkio commented 1 year ago

This is fixed in the timing branch, which should be merged soon. Closing this isssue.