search

waldronlab / GSEABenchmarkeR

Reproducible GSEA Benchmarking
Other
13 stars 4 forks source link

Is there a way to use built-in FDR control for GSEA and other methods? #14

Closed annikagable closed 4 years ago

annikagable commented 4 years ago

In the GSEABenchmarking vignette there is a section on built-in FDR control for GSEA and SAFE. The default mode of GSEABenchmarkeR seems to apply p-value adjustment for all methods in the same way, which sounds reasonable. However, with this strategy, GSEA p-values of zero (which are of course not actually zero, but GSEA outputs them as such), will not be corrected for multiple testing.

From your benchmark it doesn't look like it would make a big difference, but I was wondering whether there is a way to reproduce the data for built-in FDR control yet?

I would also be interested in the context of providing my own method with already adjusted p-values.

lgeistlinger commented 4 years ago

I don't support this anymore - given that it figured that the built-in FDR correction of these methods basically implement what stats::p.adjust with method = "BH" does.

For most benchmark scenarios, I agree that it makes sense to apply the same adjustment and note that p-values of zero remain zero independent of the adjustment method applied.

If you still wanted to try it out, you'd need to go back to the corresponding commit https://github.com/lgeistlinger/EnrichmentBrowser/commit/b14f24e14706a773fad550a15c45d78f9261f5f6. Here, EnrichmentBrowser::sbea with padj.method = "BH" invokes the built-in FDR control when using method = "gsea" or method = "safe". No guarantee that this still works with the current version of GSEABenchmarkeR, but it could still work given that runEA's padj.method argument is just passed on to sbea's padj.method argument.

I would also be interested in the context of providing my own method with already adjusted p-values.

I am not sure I follow. Where exactly (ie to which function(s)) would you like to provide results of your method with already adjusted p-values?

annikagable commented 4 years ago

If you still wanted to try it out, you'd need to go back to the corresponding commit lgeistlinger/EnrichmentBrowser@b14f24e.

Ok thanks, I might try that! At least in preranked GSEA (both in the R package and with the Java application), a pathway can have a p-value of zero but an FDR above 0.05, so internal vs external p-value adjustment are not equivalent. This would affect the number of significant sets detected in evalNrSigSets. However, this is a special case, and could also be fixed by setting the zero-p-values to 1/nr.perm.

I am not sure I follow. Where exactly (ie to which function(s)) would you like to provide results of your method with already adjusted p-values?

Never mind, I was thinking about evalTypeIError but then saw that it does not actually need any p-value correction.

lgeistlinger commented 4 years ago

Ok, let me know how this goes. If it doesn't work out, I might also be able to add it back / or create an alternative EnrichmentBrowser version for this purpose.

However, this is a special case, and could also be fixed by setting the zero-p-values to 1/nr.perm.

Agreed, or to be accurate to 1 / (nr.perm + 1) (https://www.ncbi.nlm.nih.gov/pubmed/21044043)