Performance metrics

[rnaseq]

Select various performance metrics to assess the quantification performances: e.g. % of reads that get used in quantification; sensitivity and specificity; F-measure etc.

Additionally check: RapMap paper (http://biorxiv.org/content/biorxiv/early/2015/10/28/029652.full.pdf) see section 4.2.1 and table 2.

[mw55309]

I have previously suggested for simulated data:

8) Metrics to measure – % of reads that get used in quantification; sensitivity and specificity; Others?

@rob-p has added: I suggest the ones you list above + some subset of those we adopt in the RapMap paper (http://biorxiv.org/content/biorxiv/early/2015/10/28/029652.full.pdf) see section 4.2.1 and table 2.

For real data, we could assume the latest annotation with the latest genome build is "ground truth", and see how improvements in annotation and genome have increased various metrics over time. Thoughts?

[rob-p]

Actually, I think % of reads that get used in quantification may be an interesting and illuminating metric to explore in a bit of depth. One potentially important observation we made when doing the testing in the RapMap paper is that more mapped reads does not always imply improved quantification results (at least not under all metrics). For example, RSEM uses Bowtie/Bowtie2 parameters that discard a substantial number of alignable reads (specifically, it disallows orphaned alignments). We found that this can actually lead to improved spearman correlation with the ground truth (since orphaned alignments are typically more ambiguous than paired-end alignments and often more difficult to deconvolve). However, some metrics, like the relative error w.r.t. the true number of reads, will obviously suffer when reads are discarded unnecessarily.

If we have some "higher level" notion of quality (e.g. correct DE calls), than perhaps we can get at the question of whether it matters more to keep every precious read, or to not be fooled by reads that aren't particularly informative.