2-Sample Permutation test

[Mattlab-hub]

Describe the new feature or enhancement

Please provide a clear and concise description of what you want to add or change. Hi, I would like to add a 2 sample permutation T-test to MNE. I was in a rush a while back and was unable to use the permutation_cluster_test because I could not get an adjacency matrix with the data I had. Other permutation tests were all one sample. I tried using other packages, but realized all the statistical packages for python that had a permutation test ran into one or more problems: couldn't handle nan values well, weren't invariant to the number of dimensions, or had very strict array requirements. Here is the test I threw together at the time: https://github.com/Winter-Bot/2sample_Permutation_ttest . It takes into account nan values in the ttest, it is able to take data arrays from 2D to 4D, and it can take pure arrays or a list of arrays, so long as the first dimension is the observations/participants.

Describe your proposed implementation

Describe how you think the feature or improvement should be implemented (e.g., as a new method on an existing class? as new capability added to an existing method?) If you're not sure, please delete this section and the next section. I was wondering if you wanted me to fork it. Or you could take it and modify it, adding it to other existing code. That is okay too. https://github.com/Winter-Bot/2sample_Permutation_ttest

Describe possible alternatives

If you've suggested an implementation above, list here any alternative implementations you can think of, and brief comments explaining why the chosen implementation is better. forked to mne.stats. or included into one of the other existing permutation test.

Additional comments

Add any other context or screenshots about the feature request here. I tested my code against these: (1) nnstats.permtest_rel() (Network Neuroscience Lab at McGill University, 2018) (2) mlxtend.evaluate.permutation_test() function from Raschka, Sebastian (2018) (3) mne.stats.permutation_cluster_test() function (Gramfort et al., 2014) (4) permutation_test() functions from Python Software Foundation (2017) with HCP-MEG eravg data and tfavg data. I did a short write up if you want to see it let me know. I have it give several types of outputs, each one being more or less applicable depending on the type of data being input. If you want to take it and completely reformulate it that is fine too. If you have questions for me or need me to change anything on it, you can reach me at m.a.winter87@gmail.com

[agramfort]

can you provide a code snippet that fails with mne.stats and works with your code? thx

[Mattlab-hub]

It's not that the code fails, it's that the only 2 sample permutation test is the cluster analysis (correct me if I am wrong). For me this didn't work because I couldn't get an adjacency matrix due to the data constraints. When I bypassed the adjacency matrix part the cluster analysis echoed nan values throughout the data. So if participant 1 was missing electrode 1, (due to not having an adjacency matrix I also couldn't interpolate bad channels) then the out put would have nan values for electrode one. With 83 participants this left me with an output of mostly Nan values. But no errors. So the cluster analysis is probably a better test to run, but some constraints on data may make it harder to use. At least that was what I observed, feel free to let me know if there was a better way I missed.

Best, Matt

On Wed, Feb 3, 2021, 3:01 AM Alexandre Gramfort notifications@github.com wrote:

can you provide a code snippet that fails with mne.stats and works with your code? thx

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/8832#issuecomment-772314509, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJHQ4FZ4LPPKPWLHL7ECVLS5D7GPANCNFSM4W72ZTKQ .

[drammock]

when you run permutation_cluster_test you can pass a custom stat_fun (i.e., one that handles NaNs gracefully). Some of the scipy.stats functions have a nan_policy='omit' option that might work for your use case; it might be enough to freeze that parameter using partial() --- or you may need to write a more complicated wrapper --- before passing it to stat_fun.

[Mattlab-hub]

Thanks, I didn't try all the options with the cluster analysis. I'm a bit new to MNE, so I was finding the cluster analysis I bit too much to work with. The scipy.stats functions on the other hand were too inflexible. I probably could make a better wrapper, but I wouldn't know where to begin with that in all honesty. It was just easier for me at the time to write a stripped down test that was better at handling the particular issues I was running into, being nan values and varying array dimensions. If you can't use it that's fine, just thought I'd share it incase.

On Wed, Feb 3, 2021, 1:51 PM Daniel McCloy notifications@github.com wrote:

when you run permutation_cluster_test you can pass a custom stat_fun (i.e., one that handles NaNs gracefully). Some of the scipy.stats functions have a nan_policy='omit' option that might work for your use case; it might be enough to freeze that parameter using partial() --- or you may need to write a more complicated wrapper --- before passing it to stat_fun.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/8832#issuecomment-772726758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJHQ4HVGLY3RYKBNNXJ3MLS5GKKDANCNFSM4W72ZTKQ .