brentp / fishers_exact_test

Fishers Exact Test for Python (Cython)
BSD 3-Clause "New" or "Revised" License
62 stars 21 forks source link

Results do not closely match scipy.stats.fisher_exact #18

Open allComputableThings opened 7 years ago

allComputableThings commented 7 years ago

I'm finding that this module doesn't output values that closely match scipy.stats.fisher_exact.

I this a divergence in my test. Is the scipy.stats.fisher_exact, or this library correct?

import unittest
import scipy.stats
import fisher

def fisher_twotail_scikit(contingencies):
    oddsRatio, pExact = scipy.stats.fisher_exact(contingencies)
    return pExact

import fisher # Install from: https://pypi.python.org/pypi/fisher/
def fisher_twotail(contingencies):
    ((a,b),(c,d)) = contingencies
    return fisher.pvalue(a,b,c,d).two_tail

class FisherTest(unittest.TestCase):
    def testFisher(self):

        import random
        for scale in (5, 10, 1000, 10000): # fails with scale>=1000
            for i in xrange(50):
                a = random.randint(0,scale)
                b = random.randint(0,scale)
                c = random.randint(0,scale)
                d = random.randint(0,scale)

                contingencies = \
                    [[a, b],  # outcome 0
                     [c, d]]    # outcome 1
                p1 = fisher_twotail_scikit(contingencies)
                p2 = fisher_twotail(contingencies)
                err = abs(p1-p2)
                if err >= 0.000001:
                    print contingencies
                    print p1, p2, err
                    assert err < 0.000001
tanghaibao commented 7 years ago

@stuz5000 several years back when @brentp and I implemented this, the scipy one was known to be inaccurate. It might still be the case: https://github.com/scipy/scipy/issues/4130

Best to use R to check for precision, as we did in the test. https://github.com/brentp/fishers_exact_test/blob/master/tests.py

Could you help us with some more tests with R? If results from R is different, we'll definitely look into this.