Troubleshooting For Low Matching Accuracy

[shovikb94]

Hello,

Thanks for creating this innovative tool, I think this addresses a significant gap in the spatial omics space.

I am using this approach to match data from spatial imaging (54 proteins) and scRNA-Seq. The data was acquired from adjacent sections from the same sample, so intuitively there should be matchable cells. The imaging dataset contains nearly a million cells, the scRNA-Seq contains around 100K. We are using a detailed annotation (~30 cell types).

We ran through the pipeline, but are getting a matching accuracy of ~6%. I have two questions: 1) How reliable is the matching score? I understand it is the percentage of cells with identical labels of the Modality 1 cell to the nearest neighbor by Pearson correlation in Modality 2? I get this to an extent, but does that approximate ground truth?

2) What are some troubleshooting tips for such a case? Biologically, these cells should match. I wonder whether certain parameters can be tuned, and if so which do you suggest? Are there cases you have found where this method just doesn't work well? Or, alternatively, does it not work well with more detailed annotations? We have a fairly heterogenous system.

Thank you so much for your consideration and time!

[zongmingma]

Hi, thanks for your interest. Two quick checks first:

(1) How did you normalize your data? You can see examples in our documentations on how we did it for the data analyses we presented in the manuscript.

(2) Out of the 54 proteins, how many did you find a linked transcript? What matching accuracy did you get in the initial matching?

[jonhsussman]

Thanks for the response and helping to troubleshoot and improve this tool! We used CLR normalization (across features) as implemented in Seurat to normalize the CODEX data. However, we tried it also without normalization and the difference was negligible (~6% without normalization and ~7.5% with normalization for final accuracy estimation). The RNA data was normalized with the suggested method in Scanpy as implemented in your Jupyter notebook

There were 44 matched transcripts.

The initial matching accuracy was ~9.3%.

[zongmingma]

Thanks for the clarification. I assume you have also cropped the CODEX values at the extreme percentiles. I would then check the following:

1. What was the ratio of CODEX cells against RNA (meta-)cells when you perform matching?

2. If you make UMAPs of your processed CODEX data and RNA data, are you able to differentiate the 30ish cell types in these plots?

3. If you use a coarser cell type annotation level (around 6-10 cell types) comparable to the example we give in the tutorial, what is the matching accuracy at this coarser level?

[jonhsussman]

We first filter cells by taking the sum of the markers for each cell and then remove cells where the sum of the values is <1 percentile or >99 percentile. This should have the same effect as cropping individual markers at extreme percentiles.

1. We use the default values as suggested in your Jupyter notebook for tonsil codex, so are using matching_ratio=4 and this yields 5 batches with average batch size 16,554 for RNA and 27 batches with average batch size 32,834 for CODEX. I'm not sure if this answers the question however.

2. Yes, we can distinguish the ~30 cell types using those markers in both CODEX and RNA, except for some of the smallest populations.

3. This is a good point to try sing a coarser cell type annotation, which we have done using an annotation of 7 broader cell type categories. This leads to very clearly distinguishing each cell type in the UMAP for CODEX and RNA however the accuracy does not improve much for MaxFuse. The initial lv1_acc goes to 16.5% and the final accuracy goes to 18.1%.

[zongmingma]

Hmm... this seems counter-intuitive. Have you tested other methods such as bindsc and seurat on the datasets? What do these method produce in terms of matching accuracy?

[jonhsussman]

We noticed an error in the coarse annotations, causing some to be inconsistent between the RNA and the CODEX. After correcting this error we get an accuracy of 53.3% for the initial pivot matching and then a final accuracy of 48.5%. There are a few CODEX categories that do not have an scRNA equivalent but this is a relatively small amount of the data. This is a vast improvement but still not near the 95% accuracy that is shown in the example. We have not tried other methods of cross-modality integration of this data.

[zongmingma]

These numbers sound more reasonable though still relatively low for tonsil. I would suggest the following further steps:

1. Remove the CODEX cell types that have no match in scRNAseq data and select tuning parameters carefully in the MaxFuse pipeline as illustrated in the tutorials. The default choices of tuning parameters might not be the best for your current data. As reported in one of the supplement tables, there is a negative impact on performance when one modality contains cells that have no match in the other.

2. Notice that normalization plays an important role in this pipeline. For the tonsil analysis we did, we applied separate normalizations on the linked features and on all features in the RNA assay. Details can be found here: https://maxfuse.readthedocs.io/en/latest/tonsil_codex_rnaseq.html

3. Try an alternative approach such as Seurat or BindSC, which can be used as a benchmark number. Together with the MaxFuse performance, this should give you a sense of how hard the integration task is for the current dataset pair.

We are happy to answer any further questions you may have.

[zongmingma]

As we have not heard from you for a while, we are closing this issue for now. Feel free to contact us again if you have any other questions.