optimize HDF5 reading (decoder is slow)

dune-lardon / lardon

2 stars 4 forks source link

optimize HDF5 reading (decoder is slow) #15

Open olantwin opened 3 years ago

olantwin commented 3 years ago

[x] benchmark individual parts of decoder to find bottlenecks
[x] look at maximum performance possible using pytables, h5py
[ ] fix bottlenecks

olantwin commented 2 years ago

Locally, no individual part seems to be a bottleneck. Suspicion is that it's EOS/remote I/O. EOS very heavily caches, so up to two orders of magnitude faster for cached events.

Maybe events could be prefetched/reading moved to a separate thread to not block everything else?

Not sure, why exactly this is only an issue for the hdf5 reader. Maybe the format is "too smart" and underlying assumptions break down on EOS?

olantwin commented 2 years ago

h5py is not faster, neither locally nor on lxplus/EOS