iris-hep / idap-200gbps-atlas

benchmarking throughput with PHYSLITE
6 stars 1 forks source link

feat: optimize branch selection for throughput #105

Closed alexander-held closed 3 months ago

alexander-held commented 3 months ago

The branch selection strategy previously relied on picking branches in decreasing size. Some of the large branches are also more expensive to read, so in terms of pure rate optimization that strategy is not ideal. The new strategy considers size per time to optimize for rate. As reading slows down once lots of small branches are needed, the target fraction here is 15% which seems to give a good trade-off where the rate has not dropped off too far yet, while ensuring the fraction getting read is not too small either.

resolves #25