MakeCategory.py memory leak

[jpata]

Seems like MakeCategory.py leaks memory in the map-reduce.

Specifically in https://github.com/jpata/tthbb13/blob/meanalysis-80x-makecat-lumi/Plotting/python/Datacards/MakeCategory.py#L168

ret = ... never gets cleaned up, regardless of how the histograms are created.

[jpata]

tracked down to TFile.Get("sparse"), the result of which gets NEVER gc-d (?)

test program:

import ROOT, time, resource

f = ROOT.TFile("/mnt/t3nfs01/data01/shome/jpata/tth/datacards/Sparse_Aug10.root")

for i in range(100):
    time.sleep(1)
    h = f.Get("ttHTobb_M125_13TeV_powheg_pythia8/sl/sparse")
    print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

See also this thread about the known memory usage of THnSparse.

[jpata]

Also appears in a C++ version

#include <iostream>
#include "TFile.h"
#include "THnSparse.h"

int main(int argc, const char** argv) {
    TFile* fi = TFile::Open("file:///mnt/t3nfs01/data01/shome/jpata/tth/datacards/Sparse_Aug10.root");
    for (int i=0;i<10000;i++) {
        THnSparse* h = (THnSparse*)(fi->Get("ttHTobb_M125_13TeV_powheg_pythia8/sl/sparse"));
    }
    return 0;
}

[jpata]

ROOT version

$ which root-config
/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_11/external/slc6_amd64_gcc530/bin/root-config
$ root-config --version
6.06/01

[jpata]

This is currently mitigated, but memory usage remains quite high (>2.5GB) in MakeCategory.py.