JuliaDataCubes / YAXArrays.jl

Yet Another XArray-like Julia package
https://juliadatacubes.github.io/YAXArrays.jl/
Other
99 stars 16 forks source link

Huge overhead for InDims("X", "Y") compared to loading the whole dataset into memory #297

Open felixcremer opened 1 year ago

felixcremer commented 1 year ago

I am implementing the Multiple Testing functions and I need to compute the clusters on a 15000x15000 tile. When I try to use mapCube with InDims("X", "Y") this has a huge overhead compared to loading the memory and doing the computation on the in-memory array. I don't have proper benchmarks at the moment and this is more a reminder for myself to look into it in more depth.