search

leiferlab / testPoisson

Matlab implementation of E-test for comparing poisson means. Based on Krishnamoorthy, K and Thomson, J. (2004)
2 stars 1 forks source link

Seems to be commit type 1 error... #1

Open jmossbridge opened 4 years ago

jmossbridge commented 4 years ago

Hi there,

I have been comparing the output of this script with R's Poisson.test and it seems to me your test commits type 1 error. Even in the example you give -- with one sample, there's no way the p-value should be borderline significant.

Do you know why? I have no idea, but it's just something I noticed.

Take care, Julia

aleifer commented 4 years ago

Can you clarify whether this is characteristic of the E-test as described in:

Krishnamoorthy, K and Thomson, J. (2004) A more powerful test for comparing two Poisson means. Journal of Statistical Planning and Inference, 119, 249-267]

or whether it is in fact an error in my port of their fortran code?

jmossbridge commented 4 years ago

Hi Andrew,

Sorry, I can't because there's a paywall for that paper. Do you have the original paper you can send? Then I can check.

I just know that no statistician would consider a statistical test valid with 1 observation. So maybe they never tested 1 observation and they're aware their calculations fail (but also aware that no one would ever ask to take 1 observation seriously).

More concerning is that I compared the output of your code with that of R's exact function, poisson.test, using photon absorption rates (a cononical Poisson process) and R's function gave more reasonable (larger p-value) results  -- https://www.rdocumentation.org/packages/stats/versions/3.6.1/topics/poisson.test

I wonder if Krishnamoorthy and Thomson wrote about a potential tradeoff with type 1 error, but I can't see the paper.

Anyway, wish I could help -- and happy to if you send the PDF.

Take care,

Julia

On 12/10/19 5:06 PM, Andrew Leifer wrote:

Can you clarify whether this is characteristic of the E-test as described in:

Krishnamoorthy, K and Thomson, J. (2004) A more powerful test for
comparing two Poisson means. Journal of Statistical Planning and
Inference, 119, 249-267]

or whether it is in fact an error in my port of their fortran code?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/leiferlab/testPoisson/issues/1?email_source=notifications&email_token=AEB23AVGALDPT7OZK5EXSC3QYA4JZA5CNFSM4JYCPUU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGRQCQY#issuecomment-564330819, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEB23AXLHNOFGB2ZZPD6BKTQYA4JZANCNFSM4JYCPUUQ.

aleifer commented 4 years ago

The paper is available without paywall here: https://userweb.ucs.louisiana.edu/~kxk4695/JSPI-04.pdf

Fortran and python implementations are available https://userweb.ucs.louisiana.edu/~kxk4695/statcalc/pois2pval.for and

https://pypi.org/project/poisson-etest/

There are many different statistical tests involving Poisson distributions and so I would not a priori expect the E-test to give the same results as another test.