anova significant but not post-Hoc

[aaa34169]

Hi this figure despicts my problem. i have just carried out a spm one-way anova with 3 indepedant variables (TD,PL,HL) The Ftest indicated a significant difference at frames 80 to 90 but not the posthoc test. are posthoc comparison too conservative of H0 hypothesis ? is the reason ?

regards

Fabien

[0todd0000]

Hi Fabien,

To clear up some terminology: a one-way ANOVA can only test one independent variable. That variable may have three or more different levels, but it is just one variable. Thus TD,PL and HL represent different levels of one independent variable (which might be CONDITION or GROUP, for example).

Post-hoc tests are generally more conservative than the main test unless proper procedures are employed to ensure close correspondence between the main test and the post hoc tests. The Bonferroni correction procedure which spm1d currently employs is a simple procedure, and it is possible that the results will be more conservative than the main ANOVA results.

I nevertheless agree that the results at 80-90% time appear to be strange. For the main 5-15% result we can see similar results in the (TD,PL) and (HL,PL) pairs. But the 80-90% result does not appear.

Just to check: were only these three levels (TD,PL, and HL) used in the ANOVA? From Issue #5 it seems like there might actually be six levels.

Todd

[aaa34169]

hi todd

thank you for your instructions indeed, i have one indepedant variable with 3 levels ( ie. groups : TD,PL,HL ) i tested the hypothesis : VD( group 1) = VD( group 2)=VD(group3) with VD : the depedant variable

if you want to link this post with issue #5 https://github.com/0todd0000/spm1d/issues/5. Here, i focus on One conditions ( one gait speed)

from the posthoc test, can i reconsider the difference highlight by anova ? do you plan to implement post hoc test more less conservative ?

regards

Fabien

2014-10-24 2:15 GMT+02:00 Todd Pataky notifications@github.com:

Hi Fabien,

To clear up some terminology: a one-way ANOVA can only test one independent variable. That variable may have three or more different levels, but it is just one variable. Thus TD,PL and HL represent different levels of one independent variable (which might be CONDITION or GROUP, for example).

Post-hoc tests are generally more conservative than the main test unless proper procedures are employed to ensure close correspondence between the main test and the post hoc tests. The Bonferroni correction procedure which spm1d currently employs is a simple procedure, and it is possible that the results will be more conservative than the main ANOVA results.

I nevertheless agree that the results at 80-90% time appear to be strange. For the main 5-15% result we can see similar results in the (TD,PL) and (HL,PL) pairs. But the 80-90% result does not appear.

Just to check: were only these three levels (TD,PL, and HL) used in the ANOVA? From Issue #5 https://github.com/0todd0000/spm1d/issues/5 it seems like there might actually be six levels.

Todd

— Reply to this email directly or view it on GitHub https://github.com/0todd0000/spm1d/issues/6#issuecomment-60328691.

Fabien Leboeuf

• Ingénieur de recherche au CHU de Nantes

- Docteur en mécanique-Biomécanique de l'Université de Poitiers

Pôle Médecine Physique et Réadaptation Hôpital St Jacques 85 rue saint Jacques 44 093 Nantes cedex1 --------------- Tél : 02 40 84 60 88

Port: 06 07 79 02 44 *

[0todd0000]

Hi Fabien, We do plan to implement alternative post hoc procedures at some point in the future.

However, I don't think this particular problem can be explained by an overly-conservative post hoc procedure. If you included only those three groups in the ANOVA then there should be closer agreement between the post hoc results and the main ANOVA result.

Could you copy your Python script into a reply? Maybe there is a small coding problem somewhere.

Todd

[aaa34169]

here is my code.

ttest=dict() ttesti=dict()

    alpha=0.05            
    F  = spm1d.stats.anova1( (TDdata,PLdata ,HLdata), equal_var=True )
    Fi           = F.inference(alpha)

    plt.figure() 
    Fi.plot()
    Fi.plot_threshold_label(bbox=dict(facecolor='w'))
    plt.xlabel('Time (%)', size=20)
    plt.title(r'Critical threshold at $\alpha$=%.2f:  $F^*$=%.3f' %(alpha, Fi.zstar))

    ttest["TD-PL"]      = spm1d.stats.ttest2(TDdata, PLdata, equal_var=True)
    ttest["TD-HL"]      = spm1d.stats.ttest2(TDdata, HLdata, equal_var=True)
    ttest["HL-PL"]      = spm1d.stats.ttest2(HLdata, PLdata, equal_var=True)

    nTests       = 3        
   p_critical   = spm1d.util.p_critical_bonf(0.05, nTests)

    ttesti["TD-PL"]   =  ttest["TD-PL"].inference(alpha=p_critical, two_tailed=True)
    ttesti["TD-HL"]   =  ttest["TD-HL"].inference(alpha=p_critical, two_tailed=True)
    ttesti["HL-PL"]   =  ttest["HL-PL"].inference(alpha=p_critical, two_tailed=True)

   .......
    ttesti["TD-PL"].plot(ax=ax21)
    ttesti["TD-HL"].plot(ax=ax22)
    ttesti["HL-PL"].plot(ax=ax23)   

[0todd0000]

Hi Fabien, The code looks good. The problem must lie in the dataset. If you relax the equal variance assumption (by setting "equal_var=False"), do the ANOVA and post hoc tests agree more closely? Todd

[aaa34169]

hi todd

sorry for the delay of my reply. I was in holiday for tew days. i changed the equal_var option by false

the posthoc hoc test are alway no significant ! do you want my data ?

regards

Fabien

[0todd0000]

Hi Fabien, Yes, please send your data if possible. Please send it to spm1d.mail@gmail.com. Then I'll reply here. Todd

[0todd0000]

Hi Fabien, Thanks for sending the data. I've found the source of disagreement between the ANOVA and post hoc results. The problem is that the HL and TD groups are nearly identical at ~85% time. Thus the ANOVA results reflect a PL vs. (HL + TD) comparison. If you combine the HL and TD groups into a single group, then do a two-sample t test, the results will reach significance in the area of 85% time.

In other words, ANOVA considers variance in all three groups simultaneously. The post hoc t tests consider variance only in two groups at a time. Other post hoc procedures (e.g. Tukey's, Scheffe) which do consider variance from all groups should yield better agreement with the ANOVA results. In this case those post hoc procedures will show significant differences at 85% for both the (PL vs. HL) and (PL vs. TD) comparisons.

Todd

[aaa34169]

hi todd Thank you for your response. If i well understand. the post hoc test are more conservative than anova. Do you plan to implement tukey or Scheffe tests ? if i can help you, i stay at your disposal

thank you Todd for your time

regards

fabien

2014-10-31 14:55 GMT+01:00 Todd Pataky notifications@github.com:

Hi Fabien, Thanks for sending the data. I've found the source of disagreement between the ANOVA and post hoc results. The problem is that the HL and TD groups are nearly identical at ~85% time. Thus the ANOVA results reflect a PL vs. (HL + TD) comparison. If you combine the HL and TD groups into a single group, then do a two-sample t test, the results will reach significance in the area of 85% time.

In other words, ANOVA considers variance in all three groups simultaneously. The post hoc t tests consider variance only in two groups at a time. Other post hoc procedures (e.g. Tukey's, Scheffe) which do consider variance from all groups should yield better agreement with the ANOVA results. In this case those post hoc procedures will show significant differences at 85% for both the (PL vs. HL) and (PL vs. TD) comparisons.

Todd

— Reply to this email directly or view it on GitHub https://github.com/0todd0000/spm1d/issues/6#issuecomment-61263012.

Fabien Leboeuf

• Ingénieur de recherche au CHU de Nantes

- Docteur en mécanique-Biomécanique de l'Université de Poitiers

Pôle Médecine Physique et Réadaptation Hôpital St Jacques 85 rue saint Jacques 44 093 Nantes cedex1 --------------- Tél : 02 40 84 60 88

Port: 06 07 79 02 44 *

[0todd0000]

Hi Fabien, Yes that's correct. These simple post hoc t tests are too conservative in this case. Post hoc t tests are fine when the groups differ, but since two of the groups are nearly identical the t tests become too conservative. We plan to release different post hoc options in version 0.3 which will likely be released next year. If you could test a preliminary version of those procedures we would be grateful. I'll be in touch be email soon with some preliminary code for Tukey / Scheffe procedures. Regards, Todd