search

iRNA-COSI / APAeval

Community effort to evaluate computational methods for the detection and quantification of poly(A) sites and estimating their differential usage across RNA-seq samples
MIT License
13 stars 14 forks source link

AbsQuant SWF: filter for non-zero expression #424

Closed ninsch3000 closed 1 year ago

ninsch3000 commented 2 years ago

Currently all sites reported by the participant that do not match a ground truth site are considered false positives. However, QAPA apparently reports back all sites it has encountered in its reference DB, but if it didn't find them expressed the score will be set to zero. By considering all non-matched PD sites false positives, for QAPA that number is thus overestimated. This effect is not relevant for the "core" abs quant metrics, but as we also calculate precision and sensitivity it might still influence how we represent QAPAs performance.

Whether or not we should filter for non-zero sites should be open to debate though, as one could argue that this return of all reference DB sites is a design decision of the developers, and we'd give QAPA an advantage if we filtered out those with a particular (in this case zero) score.

Opinions welcome, the implementation shouldn't be complicated, small adjustment of compute_metrics.py. We'd however have to re-run the abs quant SWF for QAPA for all challenges.

mrgazzara commented 1 year ago

In the case of AbsQuant I think just filtering out zeros reported from an EWF makes a lot of sense for the reasons you mention above @ninsch3000

Ideally this would be a SWF parameter for filtering out sites with expression less than or equal to a threshold, with a default of zero tpm.