confidence interval for an effect size in a repeated-measures ANOVA?

[dmurphy8]

* Enhancement: is there a way to get a confidence interval for effect sizes in a repeated-measures ANOVA? * Purpose: compare the magnitude of effects * Use-case: **Is your feature request related to a problem? Please describe.** **Describe the solution you'd like** You can get a confidence interval for Cohen's d in t-tests so is there a way to do it in an ANOVA? **Describe alternatives you've considered** **Additional context**

[juliuspfadt]

@dmurphy8, from what I make of this, the issue is fixed in a recent JASP version? Can you confirm this?

[dmurphy8]

No, I don’t see an option for it. How would I do it?

On Nov 8, 2022, at 6:13 AM, Julius Pfadt @.***> wrote:

@dmurphy8 https://github.com/dmurphy8, from what I make of this, the issue is fixed in a recent JASP version? Can you confirm this?

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/1299#issuecomment-1307286915, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUCLHZCE2JVH5YZYWTVGGGTWHJNXRANCNFSM444UWBDQ. You are receiving this because you were mentioned.

[juliuspfadt]

If you go to Post-hoc tests there is a box for confidence interval. Does that help?

[juliuspfadt]

This issue was fixed by https://github.com/jasp-stats/jaspAnova/pull/90.

[dmurphy8]

Yes, that’s great! But you still can’t do it for the main effects or interaction effect sizes.

On Nov 9, 2022, at 1:51 AM, Julius Pfadt @.***> wrote:

If you go to Post-hoc tests there is a box for confidence interval. Does that help? https://user-images.githubusercontent.com/38500953/200798044-c6d4f833-c779-436d-9db7-be1f8712434d.png — Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/1299#issuecomment-1308490151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUCLHZBLOGTA5TKDGR7V6Q3WHNXZRANCNFSM444UWBDQ. You are receiving this because you were mentioned.

[juliuspfadt]

@JohnnyDoorn?

[JohnnyDoorn]

Hi @dmurphy8 ,

Do you mean confidence intervals for the omega/eta effect sizes in the main ANOVA table?

Cheers Johnny

[dmurphy8]

Yes!

On Nov 9, 2022, at 7:24 AM, JohnnyDoorn @.***> wrote:

 Hi @dmurphy8 ,

Do you mean confidence intervals for the omega/eta effect sizes in the main ANOVA table?

Cheers Johnny

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[tomtomme]

This R-Package may be of help implementing this: https://easystats.github.io/effectsize/

[dmurphy8]

Awesome thanks! Happy new year!

On Dec 30, 2023, at 5:06 AM, Thomas Langkamp @.***> wrote:

This R-Package may be of help implementing this: https://easystats.github.io/effectsize/

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/1299#issuecomment-1872524937, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUCLHZGNF42CB6YNTOBWY5TYMAGVXAVCNFSM444UWBD2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXGI2TENBZGM3Q. You are receiving this because you were mentioned.

[TarandeepKang]

Hi all,

I was just coming here to suggest this. I've just seen a new paper on omega squared in Psychological Methods with advice on the best approaches for calculating confidence intervals for it, and they suggest formulae and implementations based on Ken Kelley's MBESS package so maybe give that one and the (very good) new paper a look?

Kelley, K. (2007). Confidence Intervals for Standardized Effect Sizes: Theory, Application, and Implementation. Journal of Statistical Software, 20, 1–24. https://doi.org/10.18637/jss.v020.i08 Kroes, A. D. A., & Finley, J. R. (2023). Demystifying omega squared: Practical guidance for effect size in common analysis of variance designs. Psychological Methods. https://doi.org/10.1037/met0000581 Steiger, J. H. (2004). Beyond the F Test: Effect Size Confidence Intervals and Tests of Close Fit in the Analysis of Variance and Contrast Analysis. Psychological Methods, 9(2), 164–182. https://doi.org/10.1037/1082-989X.9.2.164

[JohnnyDoorn]

@TarandeepKang and @dmurphy8 I am currently implementing some CI's for effect sizes in (RM) ANOVA, specifically the partial eta and omega effect sizes based on the effectsize package - would that suffice?

[dmurphy8]

Yes, that’d be amazing! Something just like how you can do it for the t-test would be great. Thanks!

Dillon



On Apr 3, 2024, at 1:02 PM, Johnny van Doorn @.***> wrote:

@TarandeepKang https://github.com/TarandeepKang and @dmurphy8 https://github.com/dmurphy8 I am currently implementing some CI's for effect sizes in (RM) ANOVA, specifically the partial eta and omega effect sizes based on the effectsize package - would that suffice?

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/1299#issuecomment-2035476840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUCLHZER2NJP4JUGLN5V4WDY3RN6TAVCNFSM444UWBD2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBTGU2DONRYGQYA. You are receiving this because you were mentioned.

[TarandeepKang]

Hi @JohnnyDoorn. Yes sounds great but I would definitely recommend following

Kroes, A. D. A., & Finley, J. R. (2023). Demystifying omega squared: Practical guidance for effect size in common analysis of variance designs. Psychological Methods. https://doi.org/10.1037/met0000581

because omega seems a bit of a minefield?

Best,

Tarandeep

[JohnnyDoorn]

@TarandeepKang Thanks - I took a closer look and saw that these CI methods cannot safely generalize to RM ANOVA. I asked the package maintainer for some insight on how to best resolve it. In the meantime, I will implement the CI's in the AN(C)OVA. Both MBESS and effectsize seem to use the same non-central algorithm for the CI, so we are good there.

[TarandeepKang]

Fantastic! Yes exactly, those confidence interval methods didn't seem generalisable to RM ANOVA to me either, but it's always reassuring when proper statisticians, like you, agree. :)

[JohnnyDoorn]

will leave this open for RM ANOVA CI's