Add an implementation of podio-dump in c++ to make dumping files quicker
ENDRELEASENOTES
This is an attempt at making podio-dump quicker after several complaints (e.g. https://github.com/key4hep/EDM4hep/issues/312). After some "profiling" it turns out that the slowest part in the python implementation is the loop over all the collections which can be significantly sped up by going to c++. In my local timings the current (python based) podio-dump is almost ten times slower than this (c++ based) podio-dump-cpp) for dumping the example_frame.root file from the tests (times via time)
podio-dump
podio-dump-cpp
real
12.393s
1.513s
user
8.522s
1.251s
sys
3.823s
0.296s
The main disadvantages of the c++ implementation are that we need quite a bit of boilerplate for things that are trivial in python, e.g.:
We have to manually implement argument parsing and (parts of) the tabulate functionality
Since formatting with iostream and iomanip is bordering on masochism, I have decided to pull in fmt for now. In principle c++20 has similar functionality in std::format (but no fmt::print that only comes with c++26). However, that requires gcc >=13 and clang >= 16.
Since dumping the datamodel would require quite a bit of work in c++, I would be in favor of keeping that in python in a separate tool, while the other functionality could be covered by the c++ implementation.
BEGINRELEASENOTES
podio-dump
in c++ to make dumping files quickerENDRELEASENOTES
This is an attempt at making
podio-dump
quicker after several complaints (e.g. https://github.com/key4hep/EDM4hep/issues/312). After some "profiling" it turns out that the slowest part in the python implementation is the loop over all the collections which can be significantly sped up by going to c++. In my local timings the current (python based)podio-dump
is almost ten times slower than this (c++ based)podio-dump-cpp
) for dumping theexample_frame.root
file from the tests (times viatime
)podio-dump
podio-dump-cpp
The main disadvantages of the c++ implementation are that we need quite a bit of boilerplate for things that are trivial in python, e.g.:
tabulate
functionalityiostream
andiomanip
is bordering on masochism, I have decided to pull infmt
for now. In principle c++20 has similar functionality instd::format
(but nofmt::print
that only comes with c++26). However, that requires gcc >=13 and clang >= 16.Since dumping the datamodel would require quite a bit of work in c++, I would be in favor of keeping that in python in a separate tool, while the other functionality could be covered by the c++ implementation.
TODO: