3D permutation tests (e.g. time x space x freq) is too complicated

[jona-sassenhagen]

MNE can do cluster-based permutation tests on 3D data, but from what I understand, it's fairly complicated and time-consuming to set up the connectivity matrix. Ideally, it would be super cool if we could also assume regular lattice connectivity in the second dimension and have the user only supply the spatial neighbourhood connectivity matrix. Alternatively, we should at least make the doc string a bit more explicit about what to do.

@kingjr

[mmagnuski]

+1^100 ;) In general the interface to cluster tests could be simplified, along with results (a specific object with some viz methods?) - but that would require some thought. The lattice assumption makes sense, so far I've been using fieldtrip or custom code in Julia when dealing with > 2d spaces but only rarely these spaces have more complex adjacency structure than lattice for everything apart for spatial dim (whether the spatial dim are channels or channels_by_channels etc.). One thing that I also miss in mne cluster tests are some "filtering" options like minnbchan in fieldtrip.

[christianbrodbeck]

In case it's of interest, I have cluster-based tests like this implemented at http://eelbrain.readthedocs.io/en/r-0.26/reference.html#module-testnd in a framework that was too much statistics-related overhead to add to mne-Python at the time. MEG examples at https://github.com/christianbrodbeck/Eelbrain/tree/master/examples/meg.

[mmagnuski]

Thanks @christianbrodbeck, I wasn't aware of this functionality in eelbrain (I didn't use it previously) - I'll give it a try!

[christianbrodbeck]

@mmagnuski Let me know if anything is not obvious how to do!

[cbrnr]

@jona-sassenhagen I've been struggling with cluster-based permutation tests lately (this is the first time I'm doing this so I'm a complete newbie in this area), and I find the MNE functions rather difficult to use (but mind you, FieldTrip is also not much easier with every option in the cfg struct). So any simplification is really welcome!

@christianbrodbeck this is awesome! Thanks for pointing this out, I didn't even know your package - I'll give it a try, looks nice (but I still might come back with questions :smile:).

[cbrnr]

Oh, one more thing, do you guys know how this publication/method relates to the three methods described in http://eelbrain.readthedocs.io/en/r-0.26/reference.html#module-testnd? Is this something new or is it equal to one of the existing methods?

http://www.sciencedirect.com/science/article/pii/S1053811912010300

[mmagnuski]

@cbnr threshold-free cluster-enhancement is implemented in mne (and eelbrain too IIRC) - you need to pass a dict to treshold parameter of mne.stats.permutation_cluster_test.

[christianbrodbeck]

@cbrnr TFCE is the same method (Smith and Nichols, 2009), except Mensen and Khatami seem to suggest to use e=2/3 instead of Smith and Nichols’ e=0.5. M&K mention empirical support for this in an appendix but I can’t see that appendix in the paper(?). I did not expose e and h in Eelbrain but it would be simple to do if there is a good reason for it.

[cbrnr]

@christianbrodbeck thanks, good to know that there are only minor differences.

@mmagnuski @christianbrodbeck

eelbrain requires Python '>=2.7, <3.0' but the running Python is 3.6.3

😱

I think I will continue to try MNE's TFCE.

[christianbrodbeck]

@cbrnr it's due to the tight integration with wxPython which has not made it to Python 3 yet (although it recently started beta versions), but with conda it's very painless to install.

[cbrnr]

I think wxPython 4 is already pretty usable on Python 3. But yes, I could set up a conda env with Python 2, I'm just not sure if I want to 😉 . Anyway, I'll probably come back with questions regarding the tests, thanks for the great work implementing all this!

[christianbrodbeck]

Yeah I’m planning to look into wx 4, but “pretty usable” does not sound encouraging...

On Oct 20, 2017, at 7:53 AM, Clemens Brunner notifications@github.com wrote:

I think wxPython 4 is already pretty usable on Python 3. But yes, I could set up a conda env with Python 2, I'm just not sure if I want to 😉 . Anyway, I'll probably come back with questions regarding the tests, thanks for the great work implementing all this!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[larsoner]

Have either of you tried our environment.yml?

[christianbrodbeck]

@larsoner I would prefer it not relying on development versions... there is no release build of Mayavi that works yet?

[larsoner]

Released versions of mayavi require PyQt4 which pins matplotlib down to 1.5. If you want to go that route the environment2.yml does it

[cbrnr]

@christianbrodbeck no, it really works nicely. A while ago I've ported PsychoPy to Python3, which also involved going from wxPython 3 to version 4. Everything worked almost out of the box. A few changes here and there was all it took.

@larsoner nope, MNE has always worked just fine.

[christianbrodbeck]

Thanks @cbrnr that sounds much better :) I will look into it as soon as I get around to it. I hope mayavi is compatible with wx 4 as backend. Then Qt4 should not be needed?

[jona-sassenhagen]

I think this is superseded by #4859

[christianbrodbeck]

@cbrnr in case you're still interested, the master branch of Eelbrain is now optimized for Python 3.