two-way repeated measures ANOVA and post-hoc test

[CoraBora]

Hello,

I investigate the influence of three different insoles and two different speeds on kinematic joint data (Knee, Hip, Ankle) during running. I have run a 2-way repeated measures ANOVA, where main effect of running speed and main effect of insoles is tested. First, I summarized all 6 the conditions and checked them for normality. Since not all joint data were normally distributed, I applied ANOVA for parametric data as well as for non-parametric data. I found no main effect of insoles but main effect of speed was evident for almost every joint. In order to examine the differences between the conditions more closely, I tried to run a paired t-test (I am not sure if this is correct or if I should rather use an independent t-test). With the paired t-test I get partially significant results for the main effect of insoles although I did not get them with the Anova. Now I am unsure whether the t-test is suitable as a post-hoc test or not. I’m also wondering if I need to carry out further statistical procedures such as calculate Cohen's effect size or is a t-test sufficient? I have already looked in the Matlab forum, but Todd has already said there that. Since spm1d does not officially support post hoc analyses, people should consider asking these questions in the fctSPM forum. I would be happy if you could help me.

Best regards, Cora

[tramarobin]

Hi Cora,

As your conditions are repeated measures, paired-t tests are indeed the correct solution. If you only have 2 speeds and 2 insoles, I will not do any t-test except for the interaction if it is significant.

The ANOVA is a more robust test than the paired-t test as its input is all your data, and not the subset of the data used for a t-test. In addition, and I guess that is what Todd wanted to highlight, SPM is not capable of post-hoc analysis as the t-tests assess the smoothness (or statistical distribution) of all tests independently.

What can happen is that the p-value of your t-test must be corrected to avoid increasing the type-1 error. Without correction, it is possible that the t-test is significant while the ANOVA is not. How many speed and insole conditions are you analyzing?

Having the effect size is always better as in traditional stats. If you are using fctSnPM, the function will calculate the effect sizes with Hedge’s g, as well as their 95% confidence interval on top of the 2way ANOVA and post hoc test as a form of paired t-tests corrected with Bonferroni. The paired t-tests are also corrected in regard of with the ANOVA, meaning that it synthetizes the main and interaction effects to display only the significant post-hoc regarding the results of the ANOVA.

If you wish to use the fctSnPM, you can find examples here : ...\fctSnPM\Examples There is no current examples for a 2way ANOVA with 2RM, but the closest ones are D1_ANOVA2_1rm_3x3modal.m and D2_ANOVA2rm.m.

Let me know if you have further questions.

Best, Robin

[CoraBora]

Thanks a lot for your quick reply!

I only have 2 speeds, but 3 insoles, so I thought a t-test would be a good way to find out which insoles cause significant differences in joint kinematics (if the ANOVA shows a significant main effect of the insoles).

You wrote something about the statistical distribution, so another question came to my mind. To check the normal distribution, I summarized all 6 conditions (2 velocities x 3 insoles) and used the spm1d.stats.normality.ttest function. I did this for nine kinematic joint data (ankle, knee, hip: x-y and z-direction). Some curves did not exceed the threshold at all, so for this type of data I used ANOVA for parametric data. Other curves exceeded the threshold at some points, so I thought it would be better to use ANOVA for non-parametric data in these cases. Am I right or would you do it differently?

You said that the t-tests assess the statistical distribution of all tests independently. Does this mean that I need to check each condition of the kinematic joint data separately for normal distribution before running the t-test?

However, as far as I know, I have not used a correction (unless it is already included in the code I am using). I suspect this is why some of my t-tests were significant but the ANOVA was not.

The fctSnPM function you mentioned sounds good. I will try it out and hope it works better. Also, do I need to check for normality before using the function? Because the website says that this function computes ANOVA and post-hoc tests with a non-parametric approach (permutation tests). But as I said some of my datas are normal distributed so I'm not sure if it would be the right way.

Thankyou very much for your help Robin!

[tramarobin]

With 3 modalities of insole the post hoc test is indeed needed.

The non parametric approach doesn't require a non parametric distribution and works well for normally distributed data as well. If the distribution of the data is normal, the non parametric approach gives results similar to the parametric approach. For the simplicity of the statistics of your article, I would only use a non-parametric approach.

Not using a correction for the t-tests might be the cause of their significance if the ANOVA is not significant. The code in fctSnPM automatically adjusts the p-value for the post hoc procedure.

[CoraBora]

Thankyou! Now that I've tried the function, I have another question about the SnPM plot (post hoc test; effect size: velocity). The bold blue lines are located at the significant differences. I don't understand why the t-value exceeds the threshold in some places without being blue. I thought that if the value exceeds the threshold, it means that it is significant, and since I only have two conditions (9 & 13 km/h), the t-test should give the same results as the ANOVA.

[tramarobin]

Hi Cora,

You are correct, the ANOVA and t-tests should have the same significance for the main effects when there's 2 conditions.

However, there is a part of randomness when using permutation tests. This randomnesscan be reduce by increasing the number of permutations of each test.

By default, this number is 10 times the required amount (which is 1/alpha and often be 200 as alpha is 0.05) and limited to 10000. There are no rules about how many permutations are required to have a stable results, I would say that 1000 is a good start, but the higher the better.

If you havent increased this number and 200 permutations are done, the results could be unstable. If that's the case, can you try with 10000 permutations ?

To do so, set the parameter 'multiPerm' to 500 or more. If 'maximalPerm' is not changed, the maximal number iterations of each test will be 10000. You can increase this parameter as well but the computations can become slow depending on the computer.

Also, it is important to know that the maximal number of permutation for a t-test is equal to 2 at the power of your number of participant. With 10 participants, it would be 2^10 or 1024 permutations possible.

I hope my answer helped.

[CoraBora]

Hi Robin,

I increased my permutations to 1000 and upsampled my data. Now, the results look much better. However, I still don't understand why, as in this figure, the ANOVA shows an effect right at the beginning of the curve. The post-hoc test, on the other hand, only reveals a significant difference after a short period. Where does the significance of the ANOVA at the beginning of the curve come from if it is not confirmed in the post-hoc test?

Thankyou!

[tramarobin]

Hi,

Good to hear the results are now better.

It is hard to tell with this figure, could you please send me the ANOVA of the insole effect as well as the t-test with the inference? The easiest would be to share the snpmAnalysis.m file for this analysis, so I can check the numbers instead of the graphs only.

Thanks, Robin

[CoraBora]

Yes of course. I shared the snpmAnalysis.m file in the following dropbox file https://www.dropbox.com/scl/fi/jmrdl3n3is69gsxl1njxo/snpmAnalysis.mat?rlkey=ya8olzlhka6oth88nbiprhxfd&dl=0

Thankyou!

[tramarobin]

Hi,

From what I can see for the first point of the curve, the ANOVA is just above the threshold while the post hoc is just below the threshold. As we are talking about the effect of the insole with 3 conditions, it is mostly because 2 conditions are similar ('3ANGNI' and 'Orthoback') , leading to a greater significance of the ANOVA, while when doing 2by2 comparisons with Bonferonni, just one condition make it less significant. I saw you use 1000 and 3000 permutations for the ANOVA and post-hoc (respectively), I would increase this number as values close to the threshold are sensitive to small variations.

[tramarobin]

Hi @CoraBora,

If the issue has been solved, could you please let me know or close the issue ? If you still have questions, I'll be happy to help.

Robin

[CoraBora]

I have understood that the discrepancies between ANOVA and Post-hoc tests can be attributed to the scarcity above the threshold. To address this, I have doubled the number of permutations (However, I haven't increased it further as it significantly prolongs the runtime of the script).

Attached to this email, you'll find a Dropbox link containing the results of the ANOVA analysis, which investigates the angular trajectory of another joint. Additionally, I have created a plot that visualizes both the ANOVA results (shaded in gray) and the Post-hoc test results (colorful bars at the bottom). The Post-hoc test results indicate significance when the SnPM curve surpasses the critical t-value.

Similar to our previous dataset, significant ANOVA results are present in areas where the t-test does not exhibit an effect. Moreover, there are regions where the t-test is significant, but the ANOVA is not. Would the explanation you provided previously apply to this scenario as well?

Dropbox Link: https://www.dropbox.com/scl/fo/4nyivjn518jnr47l0gzk9/h?rlkey=ydm5guvq53yp7ze4i1lzkhfhl&dl=0

[tramarobin]

Hi,

Increasing the number of permutations can be time consuming indeed. If you are satisfy with the stability it is fine.

Yes, the explanation I gave before is the same as what is happening in the new figures. For the speed effect, with only 2 conditions, the ANOVA and post-hoc are fairly consitent. For the insole effect, ANOVA is detecting potential differences between the 3 insoles, so the 2 by 2 t-tests might have different results, especially with the Bonferonni correction.

It is interesting to display these potential differences (only the ANOVA or the t-test being significant) if they have a physiological meaning - The graph you created is good to display that andcould help discuss these differences.

[CoraBora]

Okay all right that sounds good, thank you very much for your help!