Kolmogorov-Smirnov Test Simulation

[adam2392]

I fixed my simulation for the Kolmogorov in the inferential_simulation notebook by AL. However, it seems that the null also converges to 1 also.

I was just wondering if I'm doing the simulation correctly, or if this is perhaps a weakness in the test statistic? I am testing if a distribution comes from a uniform of [0,1] vs. say... normal (0,1).

Just wanted a quick 3rd person check to see if I should be looking at another test statistic.

Thanks!

[gkiar]

Looking at your plot I think that you found a great reason to perhaps not use that test statistic! Try using wilcoxon, or some other statistic? This might be a great opportunity to simply test a few and have that be a thing you learned from this inference - what tests are robust to your type of data as samples tend to infinity. :)

Greg Kiar gkiar07@gmail.com

On Wed, Feb 24, 2016 at 2:28 PM, Adam Li notifications@github.com wrote:

Assigned #6 https://github.com/Upward-Spiral-Science/the-vat/issues/6 to @gkiar https://github.com/gkiar.

— Reply to this email directly or view it on GitHub https://github.com/Upward-Spiral-Science/the-vat/issues/6#event-564201956 .

[jovo]

i'm not quite sure what you mean, but under the null, the power cannot be above alpha, no matter what (assuming the null is that F0=F1, if your null is that they are different one way, and the alternate is that they are differently different, that is different :)

if that's not clear to you, maybe re-read hypothesis testing chapter? happy to chat on thursday.

On Wed, Feb 24, 2016 at 2:44 PM, Greg Kiar notifications@github.com wrote:

Looking at your plot I think that you found a great reason to perhaps not use that test statistic! Try using wilcoxon, or some other statistic? This might be a great opportunity to simply test a few and have that be a thing you learned from this inference - what tests are robust to your type of data as samples tend to infinity. :)

Greg Kiar gkiar07@gmail.com

On Wed, Feb 24, 2016 at 2:28 PM, Adam Li notifications@github.com wrote:

Assigned #6 https://github.com/Upward-Spiral-Science/the-vat/issues/6 to @gkiar https://github.com/gkiar.

— Reply to this email directly or view it on GitHub < https://github.com/Upward-Spiral-Science/the-vat/issues/6#event-564201956> .

— Reply to this email directly or view it on GitHub https://github.com/Upward-Spiral-Science/the-vat/issues/6#issuecomment-188424814 .

the glass is all full: half water, half air. neurodata.io

[adam2392]

My null is that D is uniformly distributed and alternative is that D is say normally distributed. If what you said is true, might just be a bug in my implementation?

What hypothesis testing chapter are you referring to? Art of data science pdf, or another book?

[jovo]

i'm not sure what you mean. ks tests test the difference between 2 distributions.

if we are testing whether two groups are different, the null must have a distribution for each, and the alternative has a distribution for each.

sounds like you have only specified an alternative...

On Wed, Feb 24, 2016 at 5:15 PM, Adam Li notifications@github.com wrote:

My null is that D is uniformly distributed and alternative is that D is say normally distributed. If what you said is true, might just be a bug in my implementation?

What hypothesis testing chapter are you referring to? Art of data science pdf, or another book?

— Reply to this email directly or view it on GitHub https://github.com/Upward-Spiral-Science/the-vat/issues/6#issuecomment-188477619 .

the glass is all full: half water, half air. neurodata.io

[adam2392]

https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

I referred to that to get a basic understanding of the test. For a one sample test it tests the null hypothesis that my data comes from a reference distribution (in this case uniform)? Then the alternative is that it doesn't?

[jovo]

swing by tomorrow 1-3, i'l clarify

On Wed, Feb 24, 2016 at 7:32 PM, Adam Li notifications@github.com wrote:

https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

I referred to that to get a basic understanding of the test. For a one sample test it tests the null hypothesis that my data comes from a reference distribution (in this case uniform)? Then the alternative is that it doesn't?

— Reply to this email directly or view it on GitHub https://github.com/Upward-Spiral-Science/the-vat/issues/6#issuecomment-188534149 .

the glass is all full: half water, half air. neurodata.io

[adam2392]

Sure I'll try to come by your office after lunch.

[adam2392]

In conclusion, learned that Kolmogorov and Wilcoxon return statistic and p-value in different order.

Better print statements needed for next time :)