Currently, the interfaces for calculating and handling LD matrices stored in Zarr arrays assume that the matrices are square. This can be very expensive for extremely large matrices, when we scale to hundreds of thousands to millions of SNPs. As a way to reduce storage requirements, we can store only the upper triangular part of the matrix and reconstruct the lower triangular part on the fly when processing the data (perhaps with the help of scipy's csr_matrix?).
[x] Create utilities for converting part or all of the Zarr LD matrix to csr_matrix.
[ ] Modify iterators to allow for triangular matrices where the full data is reconstructed on the fly.
shz9
commented
5 months ago
Currently, the interfaces for calculating and handling LD matrices stored in Zarr arrays assume that the matrices are square. This can be very expensive for extremely large matrices, when we scale to hundreds of thousands to millions of SNPs. As a way to reduce storage requirements, we can store only the upper triangular part of the matrix and reconstruct the lower triangular part on the fly when processing the data (perhaps with the help of
scipy'scsr_matrix?).csr_matrix.