tomeichlersmith
commented
2 years ago I've put a lot of thought into this and I think it is a good idea to have a clear delineation on when the user should use Python-based interaction with the data files and when they should use C++-based interaction. I think the clearest separation is on filling histograms. At this point in analysis, we transition from "heavy-duty" calculations to making plots "pretty" and so I think it is a good idea to intentionally avoid implementing a C++-based histogram filling tool.
Instead, I think a Python module that helps the user fill histograms with numpy and serialize them with h5py is appropriate. This enforces the separation where C++ processors should be used to calculate new event objects while Python is used to fill histograms, merge them, and plot them.
Notice that some calculations would be classified as "analysis", but instead of enforcing a binning decision at the Cpp level, we can encourage users to calculate their final analysis variables and put those variables into the event. Then fill and plot them later like Python. In the HEP arena, many users call this "ntuplizing" where the hierarchical data is falttened in order to make python analysis easier. The method with which fire serializes hierarchical data makes it already "flattened" but users can still have Cpp processors do analysis tasks like filtering, summing, etc... and create new event objects that can be accessed by a Python plotter.
omar-moreno
I need to determine if fire should support a HistogramPool. This would significantly affect how a merging program #4 would operate and may not even be beneficial given how efficent h5py and numpy are on the analysis end.