It should be a fairly straightforward optimisation to operations that iterate over chunked sequences - IterAll, Filter, Map, etc - to parallelise derefing the meta chunks.
E.g. startup a goroutine per tuple to read in the corresponding chunk, gather results, then iterate over them. You could even go a step further and do that on every meta node in parallel.
We haven't performance tested these operations when the chunks are on disk, and when we do, I expect they'll be slowed down by linearly reading in the chunks.
kalman
commented
8 years ago
It should be a fairly straightforward optimisation to operations that iterate over chunked sequences - IterAll, Filter, Map, etc - to parallelise derefing the meta chunks.
E.g. startup a goroutine per tuple to read in the corresponding chunk, gather results, then iterate over them. You could even go a step further and do that on every meta node in parallel.
We haven't performance tested these operations when the chunks are on disk, and when we do, I expect they'll be slowed down by linearly reading in the chunks.
@rafael-atticlabs