search

Altius / hotspot2

Implementation of hotspot2 by Eric Rynes
16 stars 3 forks source link

Underflow and/or rounding issues for very significant P-values #2

Closed erynes closed 8 years ago

erynes commented 8 years ago

This does not impact hotspot calls.

This isn't visible in the browser if you view the -log10(P) and set the max y=10 or something equally reasonable.

But when a run of low P-values arises, it's possible for it to eventually oscillate between P ~ 1e-16 and P << 1e-16. (Read that as "very much less than," not a bit shift.)

Eric will look into this. Might have to use type double double.

issue02illustration_maxheight10 issue02illustration_maxheight100

erynes commented 8 years ago

Fixed by erynes in commit # 4d7401ca6070595b95d952f9709c1a01e8910570.

Changed computeStats() to algebraically refactor the P-value calculations.