Although this is primarily a research project it is good to look into possibilites for query performance optimization. One that might be most easy to implement is dask, since it has good integration with xarray and ODC. We should explore how dask can help us to improve query performance. Specific aspects to look into are:
Lazy loading. Especially of importance when you apply spatial or temporal filters after referencing a semantic concept, factbase resource or result definition. Currently data are loaded at the reference, while it is obviously better when they only get loaded after the filter is applied.
Chunking. Should improve performance e.g. for reductions on large arrays, that can easily be computed in parallel. However, I do think chunking decreases performance when the data are small. So we should in that case think of a decision mechanism that defines if data should be chunked and if yes in how many chunks. Could also be something that the user just needs to define.
Description
Although this is primarily a research project it is good to look into possibilites for query performance optimization. One that might be most easy to implement is dask, since it has good integration with xarray and ODC. We should explore how dask can help us to improve query performance. Specific aspects to look into are:
Additional context
Lazy loading with Opendatacube and Dask Parallel processing in Digital Earth Africa Dask array reduce Xarray and dask