search

AlexsLemonade / alsf-scpca

Management and analysis tools for ALSF Single-cell Pediatric Cancer Atlas data.
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Update Spatial transcriptomics benchmarking with Alevin-fry unfiltered #162

Closed allyhawkins closed 2 years ago

allyhawkins commented 2 years ago

Closes #159 and #157. This PR addresses two remaining questions for benchmarking of spatial samples:

  1. The same samples we have been using for benchmarking, SCPCR000372 and SCPCR000373, were quantified using Alevin-fry with an unfiltered permit list of all possible spot barcodes and then added to the previous comparisons of Alevin-fry knee filtering and Spaceranger.
  2. All workflows have been updated to be using an index that was generated from the same reference files so have the same possible gene list. All samples that were not already using an index generated using ensembl v104 were re-run so now all samples can be more easily compared.

After re-running the samples accordingly, I updated the analysis notebook, 13-spatial-transcriptomics-benchmarking.Rmd to display results from Alevin-fry-knee, Alevin-fry-unfiltered, and Spaceranger for both benchmarking samples. In addition to making the changes needed to show the plots with all three tools (rather than just the two included previously), I made some other additions to the anlaysis:

  1. I narrowed in on the group of genes that are appear to be "off the diagonal" in the correlation plots, plotting them by labeling them with two different colors. This allowed me to identify the criteria that specifically categorized that group of genes and then I pulled that group of genes to perform over-representation analysis. I noted that there were no significant gene lists that were enriched in that specific group of genes, with no particular pathway or gene list showing increased gene expression in Spaceranger over either of the Alevin-fry methods.
  2. I also looked at the group of genes that is uniquely quantified in Spaceranger, but not found in Alevin-fry and found again no specific pathways or gene lists that were enriched. However, when looking at the actual gene names it appears that the majority of them are ribosomal genes, mitochondrial genes, or long intergenic noncoding genes, so nothing that we would particularly be interested in keeping.

** Note that previously I had separated the ORA by sample, but I thought it made sense to only include genes that were found to have altered expression between the two tools in both samples, eliminating the two pathways that had previously been enriched in SCPCR000372 only. Let me know if you would prefer me to look at each sample individually instead.

After adding in Alevin-fry-unfiltered, I noticed that Alevin-fry was able to detect all the same spots as Spaceranger, with no loss of spots, while Alevin-fry-knee appears to lose some spots that should be there. The actual distribution of UMI/cell and genes detected/cell in alevin-fry-unfiltered appears to not be affected by this and looks to be almost identical to alevin-fry-knee. I'm struggling with another reason behind why Alevin-fry is resulting in decreased counts and genes/ cell than Spaceranger. I would expect that by using the cr-like-em resolution that we are using that we would actually be resolving some multi-mapped reads that spaceranger may not be resolving, which would result in potentially higher number of UMIs not less.

Here is the updated html of the analysis.

allyhawkins commented 2 years ago

I'm also a bit mystified by the lower numbers from alevin-fry here. I wonder if the tissue samples result in more degraded RNA, making mapping with AF less effective. This could also explain why some samples do better than others.) I don't think it is worth testing at this stage, but I wonder if this is is place where smaller kmers would make a difference.

@jashapiro I think this is a really really good point. I've noticed from experience in trying to capture regions of genes far from the 3' end that there is fall off of gene expression in spatial libraries as you move from the 3' end a lot sooner than when doing similar experiments in single-cell. It's definitely very likely that with the tissue, the RNA is not as intact.

@rob-p, we have been working on comparing quantification of spatial transcriptomics libraries with Alevin-fry to Spaceranger and have noticed that overall there is a decrease in the number of UMIs and genes detected/cell in Alevin-fry than in Spaceranger. We also have seen that there is a shift in mean gene expression in a subset of genes with this subset showing higher mean gene expression in Spaceranger. Here we are using Salmon version 1.5.2 and Alevin-fry version 0.4.1. We are mapping to the splici index and then using the cr-like-em resolution. I have tested using both the knee filtering and the unfiltered permit list and overall see very similar results with the only difference being that the unfiltered version is able to identify all spots that are identified in Spaceranger, while some of those spots are lost in the knee filtering. Both Josh and I have been struggling a bit with trying to understand why the quantification of Alevin-fry might be showing lower counts and gene expression than Spaceranger, and were wondering if this is something you had seen before? or if you have any thoughts as to why this might be the case?

One thought that we had been discussing was potentially due to the increased degradation of RNA that could be occurring in the tissue, so perhaps using smaller kmers could help resolve this. Right now we are using a kmer size of 31. Do you have any thoughts on if the kmer size would be impacting this and if so if it would be worth decreasing the kmer size? Here is a link to the html report with the findings discussed above. If you have the time, we would appreciate any feedback or thoughts that you have on this matter as you have been very helpful throughout our benchmarking process. Thank you in advance!

rob-p commented 2 years ago

Hi @allyhawkins and @jashapiro --- I probably won't have time to do a deep dive this week, as we are in the middle of finals week and I'm teaching an undergraduate class this semester. However, if you can run a fairly straightforward test, I had a thought that immediately came to mind.

If the issue is mapping-related, e.g. due to a degraded transcriptome, then there will be more reads that alevin refuses to align in selective-alignment mode, whereas CellRanger may be content taking a big soft clip. The first way I would test this hypothesis is to run alevin-fry in --sketch mode. In this case, the alignments won't be validated under some minimal required score, and so the mappings should be much more robust to degraded transcripts in such a case. If the UMI count goes up to about what we expect in --sketch mode, that is strong evidence for this hypothesis. We have a feature pending upstream in salmon that is a new and improved implementation of soft-clipping that would eventually allow recovering this in selective-alignment mode, but --sketch is the easiest way to quickly check.

The other thing you can look at is, before even running fry, the read mapping rate coming out of the salmon step. Presumably, if there are alignment quality issues due to degraded alignments, you'll see a lower overall fragment mapping rate in these samples. This is also good evidence, though not quite as strong as running in --sketch, because this will only give you insight into the alignment loss at the fragment level, not how these ultimately translate to distinct UMI counts or how they are distributed over cells.

allyhawkins commented 2 years ago

@rob-p thanks so much for the suggestion! I definitely understand that you are busy so no worries that you don't have time to go through in detail. I was able to run the samples through Alevin-fry with --sketch as you had suggested and that seems to have done the trick. It looks like the validation scoring definitely was throwing away some alignments. Thanks again for your help as always!

rob-p commented 2 years ago

Hi @allyhawkins,

Thanks for testing this out! I'm glad to see that it eliminated the discrepancy in the UMI / detection rate you observed. Also, thank you for these beautiful and detailed comparisons between methods and parameters, etc. As we (hopefully) grow our user-base for alvein-fry, these will be an invaluable resource in recommending best practices for different types of data, and also a great resource for re-running well-thought-out analyses as we make modifications and improvements to the algorithms and implementation!