Mixed ANOVA

[yschlu]

Dear Mensen

I have an urgent question. I started to get familiar with your tool. But I am still struggling in running a 3 factorial design.

I have

• 2 groups (patients vs. controls)
• 2 time points (T1, T2)
• 4 conditions

Is it correct to run it as a mixed design defining 4 groups?

group 1: patients T1 (conditions 1-4) group 2: patients T2 (conditions 1-4) group 3: controls T1 (conditions 1-4) group 4: controls T2 (conditions 1-4)

Or is there another way to do it? Any help is very appreciated!

Kind regards Yolanda

[Mensen]

Hi Yolanda,

Unfortunately the TFCE-ANOVA approach is currently only compatible with 2 factors.

This is because there are some nice computational shortcuts when running balanced designs with only 2 factors, and there are only 3 parameters to estimate (the 2 main effects and their interaction). Once you go beyond this many independent factors you start to run into both computational, as well as some theoretical issues about how to handle so many parameters, and the "correct" strategy for permutation of factors (and interactions).

In either case, I don't understand your 4 group proposal. You seem to have a 2 x 2 x 4 design, and this cannot not be compressed in any other form unless you are willing to lose some information (e.g. taking the mean of the 4 conditions). Or do you intend to have your first two factors be represented into a single factor (GroupTime), with 4 different levels? Unfortunately this could not work because you are mixing a between and a within factor into a single factor with no way to specify the relationships (and so you violate the assumption of independent observations).

What is your primary interest in your factors, do you expect interactions among all factors? You could run a few separate TFCE-ANOVAs depending on your hypotheses of interest. For example, you could run a separate analysis on each condition. Then depending on the results compare conditions to one another while keeping a time-point or group constant (or averaged).

If on the other hand you are interested in looking at the full interaction model between your factors (3 main effects, 3 2-way interactions, and 1 3-way interaction)... then I have some other more experimental scripts you could have a look at if you are both Matlab and statistics savvy.

Let me know.

[yschlu]

Hi Armand

Thank you very much for your fast response!

I am definitely interested in your additional scripts. In fact my primary interest is a 3-way interaction. Would be great If I could try to run the scripts.

I will also run few separate TFCE-ANOVAs to get a feeling for my data set. But for a publication it seems not to be the best solution due to problem of multiple comparison, right?

Kind regards Yolanda

University of Zurich Dr. phil. Yolanda Schlumpf Division of Neuropsychology Department of Psychology Binzmühlestrasse 14/Box 25 CH-8050 Zurich Switzerland

Phone: 0041-44-635-74-04 Fax: 0041-44-635-74-09 Office room number: BIN 4.D.09 http://www.psychologie.uzh.ch/neuropsy

Am 07.02.2017 um 18:46 schrieb Armand Mensen notifications@github.com:

Hi Yolanda,

Unfortunately the TFCE-ANOVA approach is currently only compatible with 2 factors.

This is because there are some nice computational shortcuts when running balanced designs with only 2 factors, and there are only 3 parameters to estimate (the 2 main effects and their interaction). Once you go beyond this many independent factors you start to run into both computational, as well as some theoretical issues about how to handle so many parameters, and the "correct" strategy for permutation of factors (and interactions).

In either case, I don't understand your 4 group proposal. You seem to have a 2 x 2 x 4 design, and this cannot not be compressed in any other form unless you are willing to lose some information (e.g. taking the mean of the 4 conditions).

What is your primary interest in your factors, do you expect interactions among all factors? You could run a few separate TFCE-ANOVAs depending on your hypotheses of interest. For example, you could run a separate analysis on each condition. Then depending on the results compare conditions to one another while keeping a time-point or group constant (or averaged).

If on the other hand you are interested in looking at the full interaction model between your factors (3 main effects, 3 2-way interactions, and 1 3-way interaction)... then I have some other more experimental scripts you could have a look at if you are both Matlab and statistics savvy.

Let me know.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Mensen]

Right, I agree that if your primary interest is in the 3-way interaction than you should probably test for that directly.

The scripts I have for that are very much in testing, but hopefully they can give you an idea of how to proceed, although I'm not sure just how much support I can give for them.

If perhaps the specific main effects are not of primary interest than taking the difference between the two time points, and running a mixed, 2-factor ANOVA on the interaction of the two remaining factors will be equivalent to the 3-way interaction (and may even be slightly easier to interpret). If that is significant you could just run a two further 2-way ANOVAs for each time point separately, to see whether those main effects also hold.

Given your primary interest in the 3-way interaction, then running the 2 2-way ANOVAs as more of a post-hoc test, (even) I wouldn't be too harsh on the multiple comparisons issue (3 total tests on highly related data). If the effects are relatively large you could adjust your threshold alpha to p < 0.016 for instance to (harshly) Bonferonni adjust for those 3 ANOVAs run... but again this is probably on the overly conservative side.

That would be a relatively straightforward approach, and you shouldn't run into too many issues and your hypothesis will be tested directly. The linear mixed model approach (the generalised method I mentioned), is not-at-all straightforward to understand intuitively, and there is no publication on this approach (yet), so you might have a tougher time selling this to reviewers.

I'd suggest trying the more straightforward (3 test) approach. If the results are unclear, or the multiplicity of testing bothers you, then we can go on and discuss the alternative approach.

Hope that's a decent answer :)

[yschlu]

Your 3 test approach makes sense to me. I will try that first.

I am using Matlab 2016a (OSX 10.10.5, Yosemite). I managed to install the mex files. But I still get an error message when I try to use the ept_ResultViewer (see below).

ept_ResultViewer FactorA peak significance found at channel 21 at sample 802: F(1) = 25.2366, p = 0.23077 FactorB peak significance found at channel 63 at sample 258: F(3) = 22.439, p = 0.019231 Interaction peak significance found at channel 41 at sample 1293: F(1) = 6.7444, p = 0.13462 Subscripted assignment dimension mismatch.

Error in ept_ResultViewer>makeERPplot (line 1125) y(i,:) = squeeze(mean(cell2mat(cellfun(@(x) x(:,IdCh,:), Data(:,i), 'UniformOutput', false))));

Error while evaluating Text ButtonDownFcn

Do you have a clue how to fix it? Are your latests scripts compatible with Matlab 2016a?

Kind regards Yolanda

Am 08.02.2017 um 15:56 schrieb Armand Mensen notifications@github.com:

Right, I agree that if your primary interest is in the 3-way interaction than you should probably test for that directly.

The scripts I have for that are very much in testing, but hopefully they can give you an idea of how to proceed, although I'm not sure just how much support I can give for them.

If perhaps the specific main effects are not of primary interest than taking the difference between the two time points, and running a mixed, 2-factor ANOVA on the interaction of the two remaining factors will be equivalent to the 3-way interaction (and may even be slightly easier to interpret). If that is significant you could just run a two further 2-way ANOVAs for each time point separately, to see whether those main effects also hold.

Given your primary interest in the 3-way interaction. And then running the 2 2-way ANOVAs as more of a post-hoc test, (even) I wouldn't be too harsh on the multiple comparisons issue (3 total tests on highly related data). If the effects are relatively large you could adjust your threshold alpha to p < 0.016 for instance to (harshly) Bonferonni adjust for those 3 ANOVAs run... but again this is probably on the overly conservative side.

That would be a relatively straightforward approach, and you shouldn't run into too many issues and your hypothesis will be tested directly. The linear mixed model approach (the generalised method I mentioned), is not-at-all straightforward to understand intuitively, and there is no publication on this approach (yet), so you might have a tougher time selling this to reviewers.

I'd suggest trying the more straightforward (3 test) approach. If the results are unclear, or the multiplicity of testing bothers you, then we can go on and discuss the alternative approach.

Hope that's a decent answer :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Mensen]

I'm also using 2016a, and don't have any issues so I assume it something with the structure of the data that is just a little different than what I'm used to. Would you be willing to send me the results file so I can try it out myself?

[Mensen]

So the problem was your Data structure was organised into rows and the script expected columns... so the cell2mat didn't work as expected.

I suppose I always used the usually columns format and never had this issue. Its now fixed though in the latest commit 115516b2da305b07ae8b8d2befc83af9b25e30bd

[yschlu]

Thanks for adapting the code!

I downloaded and installed the current version. But I still get various error messages (see below) when clicking on various buttons. I am not able to select the factor neither? Any further ideas?

…..sorry for the many messages….

Index exceeds matrix dimensions.

Error in ept_IndividualTopoplots>@(x)x(:,:,Sample) (line 10) iData = cellfun(@(x) x(:,:,Sample), Data, 'UniformOutput', false);

Error in ept_IndividualTopoplots (line 10) iData = cellfun(@(x) x(:,:,Sample), Data, 'UniformOutput', false);

Error in ept_ResultViewer>pb_IndividualTopoplots_Callback (line 464) ept_IndividualTopoplots(Data, e_loc, Results, Factor, fPoint+size(Data{1},3)*(sPoint-1));

Error in gui_mainfcn (line 95) feval(varargin{:});

Error in ept_ResultViewer (line 102) gui_mainfcn(gui_State, varargin{:});

Error in matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)ept_ResultViewer('pb_IndividualTopoplots_Callback',hObject,eventdata,guidata(hObject)) Error while evaluating UIControl Callback

Index exceeds matrix dimensions.

Error in ept_ResultViewer>pb_BarChart_Callback (line 478) Channel = Channel{1};

Error in gui_mainfcn (line 95) feval(varargin{:});

Error in ept_ResultViewer (line 102) gui_mainfcn(gui_State, varargin{:});

Error in matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)ept_ResultViewer('pb_BarChart_Callback',hObject,eventdata,guidata(hObject)) Error while evaluating UIControl Callback

Subscripted assignment dimension mismatch.

Error in ept_ResultViewer>makeERPplot (line 1092) y(i,:) = squeeze(mean(cell2mat(cellfun(@(x) x(:,IdCh,:), Data(i,:), 'UniformOutput', false)')));

Error while evaluating Text ButtonDownFcn

University of Zurich Dr. phil. Yolanda Schlumpf Division of Neuropsychology Department of Psychology Binzmühlestrasse 14/Box 25 CH-8050 Zurich Switzerland

Phone: 0041-44-635-74-04 Fax: 0041-44-635-74-09 Office room number: BIN 4.D.09 http://www.psychologie.uzh.ch/neuropsy

Am 09.02.2017 um 15:24 schrieb Armand Mensen notifications@github.com:

So the problem was your Data structure was organised into rows and the script expected columns... so the cell2mat didn't work as expected.

I suppose I always used the usually columns format and never had this issue. Its now fixed though in the latest commit 115516b

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Mensen]

Can you specify which error is for which buttons?

As far as I can remember, the individual topographies doesn't work for multiple factors (I should really disable that button when that sort of data is loaded)... [edit: never mind it does work for the main effects of multiple factors]

Oddly you are still getting the error bar problem too... which data is loaded and what do you do to get the Error in ept_ResultViewer>makeERPplot (line 1092) [Edit: I'm trying to click around trying to reproduce your errors with your data loaded, but don't get any myself, so let me know what buttons you are presses to get the errors]

[yschlu]

I systematically checked the error messages. please find attached a doc file with detailed description…...Error in ept_ResultViewer>makeERPplot (line 1092) did not occur this time….

University of Zurich Dr. phil. Yolanda Schlumpf Division of Neuropsychology Department of Psychology Binzmühlestrasse 14/Box 25 CH-8050 Zurich Switzerland

Phone: 0041-44-635-74-04 Fax: 0041-44-635-74-09 Office room number: BIN 4.D.09 http://www.psychologie.uzh.ch/neuropsy

Am 09.02.2017 um 16:28 schrieb Armand Mensen notifications@github.com:

Can you specify which error is for which buttons?

As far as I can remember, the individual topographies doesn't work for multiple factors (I should really disable that button when that sort of data is loaded).

Oddly you are still getting the error bar problem too... which data is loaded and what do you do to get the Error in ept_ResultViewer>makeERPplot (line 1092)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Mensen]

So, all those issues should be fixed in the latest version...

re: "right-click on electrode" problem turns out that if you were looking at the name of the electrode, and no just the + or . then it couldn't trace which electrode was actually selected

re: individual topoplots seems there was some very strange calculation of the sample point selected in the case where it thought there was also a 3rd frequency dimension in the data... now it may not work for the frequency domain but we'll cross that bridge when we get there

re: cluster statistics newer matlab versions didn't like the error messaging system that was implemented in the c scripts, or the commenting format... so I changed the c code and compiled for linux... but you probably need to re-mex those c-files (ept_ClusRes) for them to work.

[yschlu]

Hi Armand

Thank you very much for your fast support! I reinstalled your tool and now it runs stable. I have an additional question: What do the error bars mean (see pic attached). I just remember that in an older ResultViewer version the pic did not include them, right?

Kind regards Yolanda

University of Zurich Dr. phil. Yolanda Schlumpf Division of Neuropsychology Department of Psychology Binzmühlestrasse 14/Box 25 CH-8050 Zurich Switzerland

Phone: 0041-44-635-74-04 Fax: 0041-44-635-74-09 Office room number: BIN 4.D.09 http://www.psychologie.uzh.ch/neuropsy

Am 10.02.2017 um 12:01 schrieb Armand Mensen notifications@github.com:

So, all those issues should be fixed in the latest version...

re: "right-click on electrode" problem turns out that if you were looking at the name of the electrode, and no just the + or . then it couldn't trace which electrode was actually selected

re: individual topoplots seems there was some very strange calculation of the sample point selected in the case where it thought there was also a 3rd frequency dimension in the data... now it may not work for the frequency domain but we'll cross that bridge when we get there

re: cluster statistics newer matlab versions didn't like the error messaging system that was implemented in the c scripts, or the commenting format... so I changed the c code and compiled for linux... but you probably need to re-mex those c-files (ept_ClusRes) for them to work.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.