search

iris-hep / func_adl_uproot

Uproot-based backend for FuncADL
MIT License
0 stars 0 forks source link

avoid loading full tree into memory #3

Closed masonproffitt closed 4 years ago

masonproffitt commented 4 years ago

Trees are currently read with uproot.TTreeMethods.arrays(), which reads the entire tree into memory. This was done because lazyarrays() returns ChunkedArrays by default, which cannot be sliced in the same ways as JaggedArrays, and these kinds of slices are necessary for query selections. This method is certainly not optimal, but it works fine with relatively small ntuples. For multi-GB files, however, this will be extremely slow and likely fill up the memory and eventually crash.

masonproffitt commented 4 years ago

It looks like the lazyarrays() problem was actually fixed by https://github.com/scikit-hep/uproot/issues/412 in awkward-0.12.17, so requiring that should be sufficient to go back to lazyarrays().