Closed grst closed 1 year ago
i can help answer some of these questions (disclaimer - i still need time to familiarise myself with the new scirpy data structure but i'm familiar with the other main bits):
@gszep we talked about Repertoire metadata earlier (https://github.com/scverse/scirpy/issues/327#issuecomment-1172887702) but I think currently scverse libraries only load AIRR Rearrangements. Would love to hear that I am wrong, but I believe that to be the case.
The main thing is that the anndata.obs slot index is cell_id
, whereas the airr data's index is sequence_id
. So scirpy stores the airr data separately in the .uns
and uses the awkward arrays to both store and retrieve the information. Although i believe the awkward arrays can theoretically hold any information in the AIRR row for each contig. So you could store those information if you have it, just that it won't be retrieved automatically.
I am still learning the best practices for anndata. Say we have a usecase of 20 repertoires each with 100,000 cells or so.
In this case, would it make sense to have 20 Anndata objects and in each anndata object there would be the 100,000 'Rearrangement' rows stored under the airr path and the AIRR Repertoire stored in uns or similar?
Yes. the thing to look out for in the airr table is that your sequence_id
are unique. cell_id
can be repeated in the airr table but i think internally the individual contigs will still be ranked by umi count to sort them from largest to smallest, so that the highest expressing pairs of heavy/light or long/short contigs would be the main bcr/tcr for that cell.
Yes, I would guess keeping them separate until you want to compare them, then concatenate them as required for specific comparisons. Still an interesting scalability question as to how many samples can you practically compare in a single scirpy object.
The io
step and retrieval steps should be fast i believe?
The HDF5 file format was designed to have lazy constant time random access to any contiguous slices of your data. If these advantages are exposed in anndata.AnnData
class then we shouldn't have to worry about running out of memory and can concatenate and compare as much as we want. However as far as I can tell:
anndata.experimental.AnnCollection
and there are issues being tracked for this https://github.com/scverse/anndata/issues/793 .X
(maybe also .layers
and .obsm
?) using the backed="r"
mode but it appears that .obs
and .var
are still all loaded into memory. I've create an issue to suggest supporting lazy dataframes https://github.com/scverse/anndata/issues/981pandas
api.Thanks for all your comments!
What about Subject or Sample fields from AIRR such as the age of donor/patient or disease_diagnosis etc; metadata that would be the same for a subset or all of the cells under obs
These are usually just stored as additional columns in .obs
. Represented as categoricals, this is quite memory-efficient.
Ideally one could request an h5ad download of multiple samples and get a single h5ad file with all the GEX, Cell, Rearrangement, and Repertoire metadata embedded in that single h5ad file. Currently when you do a download you get 4 separate files, one for each type of data.
I'm definitely open to add more reader functions to load other AIRR schemas than Rearrangement
. Could you maybe point me to an example dataset that has all four data types? Then I'll take a look how to best represent them as AnnData.
I'll separately comment on scalability later.
Regarding scalability:
Different steps in the scirpy pipeline are subject to different limitations. My goal is to enable improve scalability of scirpy such that analysis of (few) millions of cells is conveniently possible on a single workstation (e.g. >200GB RAM, >30 cores). This is tracked here: https://github.com/scverse/scirpy/issues/370.
For anything >10M cells, I believe we need to move to out-of-memory and out-of-core approaches. Solutions for this are still being figured out on the AnnData side as @gszep has pointed out. AnnData's current "backed mode" does not support layers
and obsm
. I believe random access of serialized awkward arrays requires some additional effort, but will be possible.
To get a better idea of current limitations of scalability, I'll share here my experiments with omniscope's longitudonal COVID19 dataset with 8M TCR-beta chains (and no gene expression data):
index_chains
function: 2:30h (I think I can speed this up significantly using numba
, #386). ir.get.airr(adata, "junction_aa")
ir.get.airr(adata[200000:210000], "junction_aa")
The real bottlenecks of the scirpy workflow are further downstream:
I'm definitely open to add more reader functions to load other AIRR schemas than
Rearrangement
. Could you maybe point me to an example dataset that has all four data types? Then I'll take a look how to best represent them as AnnData.
Any chance you have an iReceptor Gateway account 8-)
I just did a download of the data from one subject from a cancer study from Yost et al (http://doi.org/DOI:10.1038/s41591-019-0522-3). The download contains three Repertoires from a single subject at different time points, two pre-treatment and one post-treatment. The data will have a rearrangement file, a cell file (AIRR JSON format), and a GEX file (AIRR JSON format) (as well as some other files). Each will contain all of the data of one type from all three Repertoires. That is the rearrangement file will contain rearrangements from all three repertoires (actually six repertoires in the rearrangement case, as they are split into TRA and TRB for each time point). It is a 10X study with ~4500 Cells across all three time points.
The ZIP file is 180MB so probably want to share it outside of github.
Happy to announce that the new datastructure is now rolled out as part of scirpy v0.13. Please check out the release notes.
Excellent!
Now that scirpy is part of scverse, we could think of an improved data structure for scAIRR data. See also the discussion at https://github.com/theislab/scanpy/issues/1387.
The challenge with scAIRR data is that
1
cell can haven
chains. Up to four of them are biologically meaningful but there could be more for technical reasons.The current pragmatic solution is to store all fields in
adata.obs
.adata.obs
. Also serializing excess chains is not really elegant.New options are
adata.obs
.The new representation should also aim at being a community standard for the scverse ecosystem and should build upon the AIRR rearrangement standard. Ideally, we could get additional stakeholders onboard, including conga, dandelion, tcrdist3 and possibly members of the AIRR community.