Open patgo25 opened 10 months ago
what is going on with the tcm loading!
okay thanks for testing, will check this asap.
It looks like this performance issue could be related to these: https://forum.hdfgroup.org/t/performance-reading-data-with-non-contiguous-selection/8979 and https://github.com/h5py/h5py/issues/1597
We're going to try the same workaround in read_object
in order to speed up calls when an idx
is provided.
Where do we do indexed reading in the data loader other than in .load()
, when an entry list is used?
the idx read appears to have been a red herring. I found a factor of ~3 speed up for that tcm read step (really: build entry list) in data_loader and another factor of ~2 in lgdo.table.get_dataframe. However the overall read is still a factor of ~3 slower than Patrick's low-level read:
Singularity> python kraus_speed_test.py
Try low level:
time to located files: 0.003 s
time to load meta: 1.050 s
time to load tcm: 1.632 s
time to load data: 9.426 s
Try Data loader:
time to located files: 0.726 s
time to load meta: 0.891 s
time to load tcm: 26.584 s
time to load data: 7.684 s
There is a lot of complexity in data_loader having to do with dealing with multi-level cuts and other generalizations, but I hope it can be sped up more with some refactoring. It looks like there is still a lot going on in the inner loops.
Nice! Performance of the LGDO conversion methods is also a topic for https://github.com/legend-exp/legend-pydataobj/pull/30 by @MoritzNeuberger.
Did anyone try profiling the code to spot straightforward bottlenecks?
At the analysis workshop, it was argued that the slowness of the DataLoader was only due to the IO speed. When I loaded the data with "low-level routines", I never felt it was that slow. To test this quantitatively, I wrote a quick script to compare low-level loading with DataLoader loading.
The script is:
First, I would ask you if you can find any unfair treatments in one or the other routine. Booth routines should:
The result on LNGS of the script is: