Open Moelf opened 6 months ago
Another possible way is to use StructArrays.jl more smartly, @peremato did you run into anything like this in EDM4hep.jl? If so anything you found working?
With EDM4hep, I think I do to have this problem since the top level is Vector of POD structs instead of being a struct of vectors as is in this case. It is true that I read all the fields (I guess) because I really construct at the end a SaA of the container.
Consider the following top-level field (column in the table analogy)
currently, when we loop over the events, the access is too "eager":
In this case, we only want to access the storage related to the
pTs
(i.e. rntuple column 17 and 18), but in reality we're reading all the columns (17,18,19,20,23,24,29,30,31) as soon as we doevt.var"AntiKt4TruthDressedWZJetsAux:"
One possible way is to switch to
AwkwardArray.jl
by @jpivarski, and represent the wholerntuple
as a bigRecordArray
and theoretically it will work for columnar access (i.e.rntuple.var"AntiKt4TruthDressedWZJetsAux:".pt
), and it may not solve our event-iteration problem.Another possible way is to use StructArrays.jl more smartly, @peremato did you run into anything like this in EDM4hep.jl? If so anything you found working?