ENH: mixed between and within subject contrasts with spatio-temporal clusters

[dengemann]

@yuval-harpaz and Maor are wondering how to best perform a mixed-design analysis using our clustering framework. Main question: do we need ot add code or can we just find the right way of manipultating inputs to compute the contrast. Example 4 within subject factors, 3 group factors. Non-lazy approach would be to implement a mixed-design ANOVA.

[larsoner]

I think you probably need to implement that. Or you could be statistically improper a bit and skip to the post-hoc paired or unpaired t-tests, without first doing the ANOVA to establish meaningful differences between the groups.

[kingjr]

Not sure whether this helps, but I had made a generic nesting function, to apply n-factor contrast (or regression) within each subject. The functions is available here https://github.com/kingjr/Paris_orientation-decoding/blob/master/scripts/base.py#L34

One can use it to:

• make hierarchy of evokeds at the single subject level
• then apply the spatio_temporal_cluster_1samp function across subjects

For instance, if you have subjects x (left/right) x (visual/audio), and want to test the interaction between laterality and modality the function

X = np.zeros((n_subjects, n_chans, n_times)
df = DataFrame(dict(modality=events[:, 2]<2, laterality=events[:, 2] % 2))
for subjects in subjects_epochs:
    scores, sub = nested_analysis(epochs._data, df, ['modality', 'laterality'])

would do something like

for subject, epochs in enumerate(subjects_epochs):
    data_laterality = np.zeros((2, n_chans, n_times))
    for ii, laterality in enumerate(['left', 'right']):
         data_modality = np.zeros((2, n_chans, n_times))
         for jj, modality in enumerate(['visual', 'audio']):
             evoked = epochs[laterality + '/' + modality].mean()
             data_modality[jj, :, :] = evoked.data
          data_laterality[ii, :, :] = data_modality[0, :, :] - data_modality[1, :, :]
    X[subject, :, :] = data_laterality[0, :, :] - data_laterality[1, :, :]

cluster_stats(X)

In this case, you can use X to test the highest level interaction only. You can also take the lower levels output by the function to apply lower level analyses.

The functions allows multiple parameters (specify the function applied at each level (subtraction, regression, within trials or on averages, n_jobs). The interpretative difficulty of the functions is partly due to its recursive aspect.

[dengemann]

@kingjr but this does not allow for a mixed design with repeated measures and group factor, does it?

[kingjr]

No, there's no group factor; it assumes paired tests only.

[jona-sassenhagen]

Would it be in any way feasible to make a function that can, based on MNE objects (evoked/epochs/tfr/...), transform data precisely so that it can be handed down to statsmodels? Would be really DRY.

Though I think statsmodels doesn't actually have a working ANOVA right now.

[dengemann]

No that's not an option, we can't afford a dependency on statsmodels and then their implementations are usually not optimized for mass-univariate testing. We should rather take a look at MATLAB SPM code or so.

[jona-sassenhagen]

Are you aware of the Massive Univariate Toolbox by David Groppe and LIMO EEG by Pernet & Rousselet (both for EEGLAB)? Maybe they could also be helpful. Groppe & Rousselet really know their stuff.

[kingjr]

It's on Github, and the functions are documented https://github.com/dmgroppe/Mass_Univariate_ERP_Toolbox => @dmgroppe I am not clear of the licence though, so I'm not sure whether we can directly translate their functions in python.

I haven't found a mixed-model analysis, but I haven't done an exhaustive search.

[jona-sassenhagen]

I don't think David's toolbox does between designs. LIMO probably does though.

David would probably be willing to relicense, he was very responsive and friendly when I wrote him some time ago. LIMO is LGPL, but they'd probably still relicense also.

[dmgroppe]

Hi Jean-Remi and Jona, The Mass Uni Toolbox does not do mixed-model analysis. It just does one-sample, paired-sample, and independent sample t-tests. I believe LIMO does do mixed-model analyses. I'd be happy to contribute the MUT to MNE-Tools and am psyched to see some people finally making a well-developed package for EEG/MEG in Python. What kind of license do you need it to have? cheers, -David

On Wed, Oct 28, 2015 at 7:36 PM, jona-sassenhagen notifications@github.com wrote:

I don't think David's toolbox does between designs. LIMO probably does though.

David would probably be willing to relicense, he was very responsive and friendly when I wrote him some time ago. LIMO is LGPL, but they'd probably still relicense also.

— Reply to this email directly or view it on GitHub https://github.com/mne-tools/mne-python/issues/2572#issuecomment-152031187 .

[dengemann]

Thanks David!

We need a BSD compatible license, MNE is on BSD-3. In other words it cannot be GPL.

On Thu, Oct 29, 2015 at 3:13 AM, David Groppe notifications@github.com wrote:

Hi Jean-Remi and Jona, The Mass Uni Toolbox does not do mixed-model analysis. It just does one-sample, paired-sample, and independent sample t-tests. I believe LIMO does do mixed-model analyses. I'd be happy to contribute the MUT to MNE-Tools and am psyched to see some people finally making a well-developed package for EEG/MEG in Python. What kind of license do you need it to have? cheers, -David

On Wed, Oct 28, 2015 at 7:36 PM, jona-sassenhagen < notifications@github.com> wrote:

I don't think David's toolbox does between designs. LIMO probably does though.

David would probably be willing to relicense, he was very responsive and friendly when I wrote him some time ago. LIMO is LGPL, but they'd probably still relicense also.

— Reply to this email directly or view it on GitHub < https://github.com/mne-tools/mne-python/issues/2572#issuecomment-152031187

.

— Reply to this email directly or view it on GitHub https://github.com/mne-tools/mne-python/issues/2572#issuecomment-152045405 .

[dmgroppe]

Thanks Denis. I rebooted the repo with a BSD-3 license: https://github.com/dmgroppe/Mass_Univariate_ERP_Toolbox

Let me know if you need anything else. cheers, -David

On Thu, Oct 29, 2015 at 3:12 AM, Denis A. Engemann <notifications@github.com

wrote:

Thanks David!

We need a BSD compatible license, MNE is on BSD-3. In other words it cannot be GPL.

On Thu, Oct 29, 2015 at 3:13 AM, David Groppe notifications@github.com wrote:

Hi Jean-Remi and Jona, The Mass Uni Toolbox does not do mixed-model analysis. It just does one-sample, paired-sample, and independent sample t-tests. I believe LIMO does do mixed-model analyses. I'd be happy to contribute the MUT to MNE-Tools and am psyched to see some people finally making a well-developed package for EEG/MEG in Python. What kind of license do you need it to have? cheers, -David

On Wed, Oct 28, 2015 at 7:36 PM, jona-sassenhagen < notifications@github.com> wrote:

I don't think David's toolbox does between designs. LIMO probably does though.

David would probably be willing to relicense, he was very responsive and friendly when I wrote him some time ago. LIMO is LGPL, but they'd probably still relicense also.

— Reply to this email directly or view it on GitHub <

https://github.com/mne-tools/mne-python/issues/2572#issuecomment-152031187

.

— Reply to this email directly or view it on GitHub < https://github.com/mne-tools/mne-python/issues/2572#issuecomment-152045405

.

— Reply to this email directly or view it on GitHub https://github.com/mne-tools/mne-python/issues/2572#issuecomment-152099933 .

[jona-sassenhagen]

LIMO can do between designs, there's an example in the paper.

[larsoner]

Shall we close this? Not sure there is much to change at the MNE end

[kingjr]

IMO we need to find a systematic approach a la R and add the corresponding example

On Tuesday, 18 April 2017, Eric Larson notifications@github.com wrote:

Shall we close this? Not sure there is much to change at the MNE end

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/2572#issuecomment-294827678, or mute the thread https://github.com/notifications/unsubscribe-auth/AEp7DKv5ECU2_epXYBA8W66A5nzSfLzuks5rxLEggaJpZM4GXLIi .

[larsoner]

So do something like in this example:

https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_stats_cluster_spatio_temporal_repeated_measures_anova.py

But explicitly suggest that people use e.g. statsmodels to set up their data such that conditions are exchangeable under the null hypothesis? statsmodels can do mixed within- and between-subjects designs now I think.

[jona-sassenhagen]

Somewhat related: https://github.com/statsmodels/statsmodels/pull/3303

[kingjr]

@Eric way too many lines in this tutorial. There should be a simpler API

On 18 April 2017 at 10:02, jona-sassenhagen notifications@github.com wrote:

Somewhat related: statsmodels/statsmodels#3303 https://github.com/statsmodels/statsmodels/pull/3303

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/2572#issuecomment-294854126, or mute the thread https://github.com/notifications/unsubscribe-auth/AEp7DFaGwAhPedjUBL_lHuL7E1SSIIaMks5rxMJkgaJpZM4GXLIi .

[larsoner]

This has been superceded by #4859