search

MinervaExpt / CCQENu

A port of the CCQENu package from the MINERvA offline software framework to Ben's New Systematics Framework
1 stars 3 forks source link

Address memory use in analyzer #6

Open hschellman opened 2 years ago

hschellman commented 2 years ago

Lots of samples and variables leads to a huge memory footprint.

Suggest read the file to map the contents. That just requires the keys. Store the histogram names in a map.

Then get them as needed in the extraction, sample by sample and variable by variable. Drop when no longer used.

ie std::cout << " just before 2D loop" << std::endl; for (auto samples:hists2D){ std::string sample=samples.first; for (auto variables:hists2D[sample]){ // only do this for a subset to save output time. std::string variable = variables.first; std::string basename = "h2D"+sample+""+variable; hists2D[sample][variable] = LoadMe(sample, variable); int exit = GetCrossSection(sample,variable,basename,hists2D[sample][variable],response2D[sample][variable],config,canvas2D,norm,POTScale,h_flux_dewidthed,unfold,num_iter,DEBUG); DropMe(sample, variable); if (DEBUG) std::cout << exit << std::endl; } }

Where LoadMe and DropMe do the Get and then Delete on the relevant histograms

nhvaughan commented 2 years ago

Maybe move plotting functions outside of the actual analysis operations. This would start by removing the PlotCVAndError and PlotErrorSummary functions from GetXSec.h. Since we write the hists to file at each step of the analysis, they could be retrieved in a different part of the C++ code (either GetXSec.h or analyze), or accessed separately using a python script.

hschellman commented 2 years ago

addressed partially with analyze_v8 which takes a sample name as a 3rd argument