Enforce canonical for seurat conversion.

[Bento007]

Motivation

To convert an anndata dataset to seurat we need to enforce canonical form of the sparse matrixes. Because of how the anndata library read and write the sparse format, the whole dataset must be read into memory. This is an expensive operation that is slow and memory Intensive. Since this enforcement is only required for seurat conversion, the step of enforcing canonical should be move to the seurat conversion container. This will allow us to speed up validation.

Definition of Done

• canonical form is enforced only for datasets that support seurat conversion
• canonical form is enforced in the same AWS batch job the seurat conversion happens.

Tasks

• remove canonical enforcement from the cli
• enforce canonical form in the seurat container before converting to seurat.

[nayib-jose-gloria]

Estimate (incl testing): 2-4 days

@Bento007 for comment if you agree

[nayib-jose-gloria]

From @ivirshup:

Enforcing canonical format is the biggest consumer of memory because it requires reading in the whole X and raw.X matrix.

Btw, this should be possible without reading the whole matrix, since canonicalization should only remove entries. That part shouldn't matter if the canonicalization isn't happening in place though. I can advise on this if needed.

Recommend contacting him for more details to see whether we can implement canonicalization w/o reading in the whole matrix even in the seurat conversion