DISCUSS: Stats docs

[larsoner]

People seem to have trouble using our stats code. I think the biggest problem is that our explanations are not so clear. Maybe there are some API simplifications we could make, too, but hopefully rewriting docs will make this clear.

I think we need to make a single tutorial file that has:

1. basic text about stats:
   • null hypothesis
   • parametric statistics (t-test, ANOVA)
   • non-parametric resampling methods
   • multiple comparisons / FWER / FDR
2. for each step / example, integrate toy examples with lists/NumPy arrays of different sizes (like I asked @cbrnr to do for #4731)
3. have a big table of our existing functions with suggestions for how / when to use them
4. add links to specific examples that use some of these

I think this would go a long way toward resolving some ambiguity for people.

We currently have the following examples / tutorials for stats -- I think that the permutation/clustering one is too long / hard to navigate (can probably be collapsed to 3-4 examples after the above tutorial is done):

• FDR:

  • https://github.com/mne-tools/mne-python/blob/master/examples/stats/plot_fdr_stats_evoked.py

• Linear regression

  • https://github.com/mne-tools/mne-python/blob/master/examples/stats/plot_linear_regression_raw.py
  • https://github.com/mne-tools/mne-python/blob/master/examples/stats/plot_sensor_regression.py

• non-parametric / permutation-based:

  • https://github.com/mne-tools/mne-python/blob/master/examples/stats/plot_cluster_stats_evoked.py
  • https://github.com/mne-tools/mne-python/blob/master/examples/stats/plot_sensor_permutation_test.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_1samp_test_time_frequency.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_methods.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_spatio_temporal.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_spatio_temporal_2samp.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_spatio_temporal_repeated_measures_anova.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_time_frequency.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_time_frequency_repeated_measures_anova.py
  • https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_spatio_temporal_cluster_sensors.py

Thoughts?

[agramfort]

you have my blessing :)

[larsoner]

anything missing from the list for you? linear regression should probably be in the tutorial, too, but I'd rather let someone else add that (I have limited experience using it)

[jasmainak]

+1000 me and Jaakko had a lot of trouble identifying the right functions to do stuff. There were too many functions which seemed similar to each other.

[cbrnr]

This is great! Integrating examples with a coherent tutorial text as you suggest would really improve the stats module a lot. I also like the idea to clean up the examples after the general document is done.

As @jasmainak points out, there is a lot of overlap between several functions, especially in the permutation cluster module. For instance, why can't there be just one permutation_cluster function instead of having permutation_cluster_test, permutation_cluster_1samp_test, spatio_temporal_cluster_test, and spatio_temporal_cluster_1samp_test? The function could decide whether to perform a one sample test vs. a two sample test based on the length of the first argument list. Adding to the confusion, permutation_t_test sounds as if it belongs into the same category, but it does not - maybe some reorganization could help clear things up a bit. Also, bonferroni_correction and fdr_correction could be merged into one function (R has a p.adjust function for that). There's certainly more, but these are the most important points that come to mind (because these are the functions I've worked with so far).

[mmagnuski]

I agree with @cbnr - there is a need for a more general, more user-friendly interface to cluster-based permutation tests. I have frequently found people (students for example) confused by the current way clustering tests work. It would be really cool if we could have and interface where one could do this for example:

clst = ClusterTest([evokeds_cnd1, evokeds_cnd2], test='independent t-test', adjacency=adj,
                   tmax=0.5, picks=chan_idx)
# and ideally something like:
clst.pick_clusters('p < 0.05')
clst.plot_clusters()
# which would then raise an error because no clusters were left ;)

That's a far fetched vision, but at least the first step could be done in not so distant future - first for simple cases like evokeds with t-tests and later anova or regression (I have a fast cluster_based_regression that I could contribute to mne). As a small step in this direction in the next week I can finally add a plotting function for adjacency matrices.

[agramfort]

I like the idea of drafting the API and start by how examples would be simplified showing short and possibly incomplete code snippets

[jona-sassenhagen]

Joint clusters test function would be cool.

As would cluster_based_regression.

For the people I hang out with, a thing they do a lot is select mean activity in some window of activity (mean time points, usually over trials, and sometimes channels) and then run an ANOVA or mixed-effects regression over subjects and conditions. I'd contribute a write-up for that eventually.