Use of SciPy resampling/MC functions?

[mdhaber]

@stefanv In Seattle and @matthew-brett in Santa Rosa, we discussed that SciPy has functions for performing

• Monte Carlo hypothesis tests (scipy.stats.monte_carlo_test),
• permutation hypothesis tests (scipy.stats.permutation_test), and
• the bootstrap (e.g. confidence intervals, scipy.stats.bootstrap).

There is also a convenience function for Monte Carlo goodness-of-fit tests (scipy.stats.goodness_of_fit).

The functions are featured in a special section of the stats tutorial ("Resampling and Monte Carlo Methods"), and some of SciPy's hypothesis test functions have examples showing how the results can be reproduced and/or extended with a permutation or Monte Carlo approach (e.g. scipy.stats.spearmanr). We've also paved the way for adding a method keyword argument to existing functions, which simplifies the generation of permutation or Monte Carlo p-values (e.g. scipy.stats.anderson_ksamp) and bootstrap confidence intervals (e.g. scipy.stats.pearsonr) using that function's statistic.

I saw in source/about_technology.Rmd that there is some intent to use SciPy in the book, but a search on GitHub didn't find examples. Does an introduction to these functions belong in the book? If so, as the author of these functions, can I help? Thanks for considering it!

[matthew-brett]

So sorry to be slow to get back to this one. I think that we have got to the stage where we have a book that usable, and is largely a port of the original book, but with lots of plans to extend and refactor the book for a subsequent addition. For that book, we do want to discuss things like t-tests, and automating permutation tests and so on - so the Scipy routines would be a good fit. I'll leave this one open to remind us.

[mdhaber]

OK, thanks for keeping it in mind! Thought I'd mention that since writing this, I also added scipy.stats.power for simulating the power of a test w.r.t. an alternative. We also added executable notebooks for several hypothesis tests, about half of which show what is going on under the hood and how an analogous permutation or Monte Carlo version can be performed.

[matthew-brett]

Nice - thanks for the pointer - I'll have a look at the power stuff, it sounds useful.