search

zrcjessica / demux

A Snakemake pipeline for running single-cell demultiplexing simulations.
2 stars 0 forks source link

a script for summarizing simulation statistics #3

Closed aryarm closed 4 years ago

aryarm commented 4 years ago

We need a script to run at the end of the pipeline after demuxlet. It should calculate summary statistics for the results of the simulation:

zrcjessica commented 4 years ago

I'm not sure if this simulation is best suited to answer the question of ideal number of animals to use per experiment - in the case of demultiplexing, less is better (considering the probability that a doublet contains cells from different individuals scales with the number of samples). I think this question is meant to be answered by the power analysis.

aryarm commented 4 years ago

yeah, I was also thinking that. plus, we currently only have 5 samples

Maybe it would be good to calculate Cohen's Kappa and some of the other metrics in that paper I gave in journal club?

aryarm commented 4 years ago

ok, I've made some progress on this here are some initial results:

precision/recall:
                   DBL           SNG
precision     0.864975      0.949100
recall        0.698375      0.975775
fscore        0.772798      0.962253
support    4247.000000  24479.000000

hamming loss:
{'DBL': 0.02062632446432776, 'SNG': 0.006250255320887291, 'BOTH': 0.00837568753046021}

The precision/recall for the doublets isn't really that high. This seems to indicate to me that there are a good number of doublets that are being predicted as singlets. And yet the hamming loss for the doublets is really low, which seems to indicate that when doublets are falsely predicted as singlets, demuxlet is often nonetheless pretty good at predicting at least one of the samples.

This got me thinking: maybe the falsely classified doublets are those with the same sample? agh so maybe the doublet scores are artificially deflated because of issue #1? Update: initial evidence seems to support my hypothesis

aryarm commented 4 years ago

ok Graham thought the results looked good, but he wants a precision/recall curve for the singlets, so we need to add that to the script