Try sparse arrays in Dask prototypes

related-sciences / gwas-analysis

GWAS data analysis experiments

Apache License 2.0

24 stars 6 forks source link

Try sparse arrays in Dask prototypes #8

Open eric-czech opened 4 years ago

eric-czech commented 4 years ago

This isn't supported in Zarr yet but because it is in Dask and the majority of genotyping data sets have major alleles with high frequency, there may be some significant speedups resulting from a sparse in-memory representation.

hammer commented 4 years ago

For reference here's the Zarr spec issue tracking support for sparse layouts: https://github.com/zarr-developers/zarr-specs/issues/48