Step-by-step benchmarking plan

[FFinotello]

Hi @mlist,

I opened this issue to coordinate the next steps in our benchmarking study.

I'd say the first important decision to take regards the single-cell data we use (for signature building and pseudo-bulk simulation). I would propose three datasets:

• A well-annotated human PBMC dataset like the Hao one. Pros: 1) Easy to annotate in terms of major immune cell types or more fine-grained subtypes; 2) availability of two real bulk PBMC datasets for testing, where 1st-gen methods obtained very good performance. Lorenzo has started to look into the Hao dataset.
• Lung-cancer atlas: Pros: we have plenty of options to test the cross applicability of signatures considering different single-cell technologies, tissue and disease contexts etc. while having common labeling of cell types. We can select data subsets for specific evaluations.
• A non-human dataset like the Tabula Muris: Pros: 1) Smart-seq2+10x data from (almost) the same cell types; 2) immunedeconv-based methods can be used as "baseline" for immune cells; 3) availability of real, murine bulk RNA-seq data for testing. We can select one or more tissues. Cons: cell type labels are often too coarse (e.g. T cells but not CD8 and CD4) or do not match between tissues. Shall we integrate, preprocess, and re-annotate this data ourselves or is there any independent effort already close to a unified and fine-grained reannotation of this dataset (e.g. Sfaira's team)? We could also consider alternative single-cell datasets of murine immune cells; any ideas?

Looking forward to your thoughts.

Meanwhile, I tag @grst @alex-d13 and @LorenzoMerotto who can provide valuable input.

Ciao, Francesca

[grst]

I agree that this is a good selection. Depending on the use-case you could also consider the HLCA instead of my LuCA. Might also be an option to use one for signature generation and the disjunct datasets from the other for validation.

Differences

HLCA

• only 10x data (but a mix of different chemistry versions)
• even more fine-grained cell-type annotation, and a 5 different hierarchy levels for cell-types
• various anatomical regions from the lung
• only healthy lung

LuCA

• 6 different platforms
• mostly lung cancer patients, matched tumor/normal in many cases
• neutrophils
• 3 hierarchy levels for cell-type annotation.

The healthy controls from LuCA are almost all also part of the HLCA.

[FFinotello]

Hello @grst, thank you so much for your input!

I think that HLCA also cover a few diseases and has additional technologies in the extended data. @LorenzoMerotto can say more about that

But you are right that the "neutrophil" aspect is valuable for our analyses (e.g. simulation with mRNA bias)! Which datasets/studies in LuCA have the highest absolute number (not percentage) of neutrophils? And on which technologies were they based? For our benchmarking, I would focus on 10x and Smart-seq2 only.

Meanwhile... https://twitter.com/ScienceMagazine/status/1524812952712491013 :)

[grst]

Indeed, i was referring to the core atlas.

It seems it's the week of atlases :) Great stuff!

[LorenzoMerotto]

The HLCA contains data fom many type of diseases in the extended section, but it seems that they are not 'officially' annotated, meaning that we could re-annotate them ourselves using the core (with all its limitations). Speaking of the technologies included, as @grst said the majority of the data is 10x, with only a few datasets in the extended version sequenced with Dropseq/Seqwell

[LorenzoMerotto]

@grst I'm leaving this question here since I don't want to open a new issue. In the LuCA atlas there are both tumor and not tumor samples. However even in those datasets where all the patients are healthy (non-cancer label), such as the Travaglini dataset, there are some cells labeled as 'Tumor cells' How should we treat that annotation?

[grst]

These would be cells that cluster (for whatever reason) with the tumor cells from other datasets. For the deconvolution benchmark I would treat them as artifact and exclude them.