GenABEL-Project / OmicABELnoMM

OmicABELnoMM implements a high-performance computing algorithm for linear regression of omics data
GNU General Public License v3.0
1 stars 3 forks source link

Large p-values incorrectly approximated #2

Open avsmith opened 7 years ago

avsmith commented 7 years ago

In the file Algorithm.cpp, p-values for T less than 1.6 are being approximated by the following (likely for speed reasons):

                if( t < 1.6)
                {
                    t1=4.4-t;
                    ptemp=0.5-0.1*t1*t;
                }
                else
                {
                    ptemp=erfc(t*one_oversqrt2);
                }

However, the approximated p-values for t<1.6 are too high. Plotting p vs t results in a discontinuity at 1.6.

lckarssen commented 7 years ago

Thanks for reporting this and providing a fix! I'll have a look at it next week, trying to see if I can find impact on speed.