search

cms-tau-pog / TauFW

Analysis framework for tau analysis at CMS using NanoAOD
9 stars 40 forks source link

Fix parallel processing for python3 #52

Closed IzaakWN closed 8 months ago

IzaakWN commented 8 months ago

Fix parallel processing, which causes segmentation faults in python3.

This is caused due to a conflict between how python3 and ROOT handle objects in the memory. The segmentation fault happens when manually deleting a ROOT histogram that was piped from a different thread created by python.

The reason why I previously implementation the manual deletion of histograms, it because in the past I was getting a segmentation fault caused when too many histograms are created and left lingering in the memory. The main way to combat this was to clean the memory by manually deleting unneeded histograms, and closing/reopening the (input) ROOT files to which the histograms were assigned. I suspect it is not as much of a problem anymore in new python and ROOT versions, but we should keep this in mind if segmentation faults happen again. Hopefully switching the plotting routine to RDataFrame will automatically solve this memory issue (see Issue https://github.com/cms-tau-pog/TauFW/issues/51), because of different way it handles files, and because we do pipe them anymore via python.

IzaakWN commented 8 months ago

@oponcet & I validated the output. It solves the segmentation faults. However, the output histograms that are produced with parallel processing do not seem to display nicely by Visual Studios from the ROOT file for some reason. It looks fine in the TBrowser.

Will merge now.