add `mask_obs`/`mask_var` arguments where appropriate

ivirshup commented 2 years ago

I think we should introduce a standardized “mask” argument to scanpy functions. This would be a boolean array (or reference to a boolean array in obs/ var) which masks out certain data entries.

This can be thought of as a generalization of how highly variable genes is handled. As an example:

sc.pp.pca(adata, use_highly_variable=True)

Would be equivalent to:

sc.pp.pca(adata, mask="highly_variable")
# or
sc.pp.pca(adata, mask=adata.obs["highly_variable"])

One of the big advantages of making this more widespread is that tasks which previously required using .raw or creating new anndata objects will be much easier

Some uses for this change:

Plotting

A big one is plotting. Right now if you want to show gene expression for a subset of cells, you have to manually work with the Matplotlib Axes:

ax = sc.pl.umap(pbmc, show=False)
sc.pl.umap(
    pbmc[pbmc.obs["louvain"].isin(['CD4 T cells', 'B cells', 'CD8 T cells',])],
    color="LDHB",
    ax=ax,
)

If a user could provide a mask, this could be reduced, and would make plotting more than one value possible:

sc.pl.umap(
    pbmc,
    color=['LDHB', 'LYZ', 'CD79A’],
    mask=pbmc.obs["louvain"].isin(['CD4 T cells', 'B cells', 'CD8 T cells’,]),
)

Other uses

This has come up before in a few contexts:

Performing normalization on just some variables https://github.com/scverse/scanpy/issues/2142#issuecomment-1046729522
Selecting a subset of variables for DE tests: https://github.com/scverse/scanpy/issues/1744
- See also https://github.com/scverse/scanpy/issues/748
Changing use_raw https://github.com/scverse/scanpy/issues/1798#issuecomment-819998988

Implementation

I think this could fit quite well into the sc.get getter/ validation functions (https://github.com/scverse/scanpy/issues/828#issuecomment-560072919).

flying-sheep commented 5 months ago

@Intron7 said he had experience with this and it’s a really good way to do things fast with dask etc.

works well in dask
scale is a good example how to do it
can’t do in 3rd party like PCA

flying-sheep commented 3 months ago

@Intron7 still waiting for your comment here!

Intron7 commented 3 months ago

Yes, we already have a good mask for sparse scaling. Boolean arrays are very effective for indicating where computations should be performed, as they eliminate the need for copying and reintegration.

One clear example is the tl.score_genes function. masks there as booleans for the nanmean is a lot more efficent but less pythonic

flying-sheep commented 3 weeks ago

Where does a mask make sense:

where is a mask better than just subsetting and then calling the function on the subset?
where is a use case for doing it on a subset (e.g. probably none for qc, right?
things that create an obsm/obsp entry would result in empty rows, but would there be a use case?

Function	Should have mask	Has mask	Notes
`pp.calculate_qc_metrics`	N	N
`pp.filter_cells`/`pp.filter_genes`	N	N	returns mask
`pp.highly_variable_genes`	N	N	returns mask
`pp.log1p`	Maybe	N	because scale has it?
`pp.normalize_total`	Maybe	N	because scale has it?
`pp.regress_out`	N	N
`pp.scale`	Y	Y
`pp.subsample`	Y	N	maybe as `weighting`[^1] also add `axis` arg
`pp.downsample_counts`	N	N
`pp.recipe_*`	N	N
`pp.combat`	Y	N
`pp.scrublet`/`pp.scrublet_*`	Maybe	N	maybe gene space
`pp.neighbors`	Maybe	N	creates obsp entry in relation to subset, like `.fit(X).transform(Y)`
`pp.pca`	Y	Y	has `mask_var`
`tl.tnse`/`tl.umap`/`tl.diffmap`/`tl.draw_graph`	Maybe	N	creates obsm entry maybe as data source instead of `n_pcs`
`tl.embedding_density`	N	N
`tl.leiden`/`tl.louvain`	Y	Y	as `restrict_to`[^2]
`tl.dendrogram`	Y	N	probably useful to make dendrograms for multiple subsets?
`tl.dpt`	Maybe	N	Ask people?
`tl.paga`	N	N
`tl.ingest`	Maybe	N	might be useful to use only part as reference? maybe not.
`tl.rank_genes_groups`	N	N	`groups`+`reference` fullfills that purpose
`tl.filter_rank_genes_groups`	N	N	creates a mask
`tl.marker_gene_overlap`	N	N
`tl.score_genes`	N	N	uses it internally, so probably don’t expose but refactor to use it[^3]
`tl.score_genes_cell_cycle`	N	N
`get.aggregate`	Y	Y

[^1]: column types: bool: subset; numeric: rel. weight per observer; cat: biased sampling [^2]: unify restrict_to and mask? [^3]: todo: refactor score_genes

scverse / scanpy