search

jasp-stats / jasp-issues

This repository is solely meant for reporting of bugs, feature requests and other issues in JASP.
58 stars 29 forks source link

Incorrect CI bars for repeated measures ANOVA (standard and Bayesian) #1187

Closed Angalanse closed 6 months ago

Angalanse commented 3 years ago

standard RM ANOVA Bayesian RM ANOVA SPSS RM ANOVA

JohnnyDoorn commented 3 years ago

Hi @Angalanse,

Thanks for your report. What options did you use for the plots? For the frequentist RM ANOVA, you can either plot by CI or standard error, which can lead to different sized intervals. Also, have you averaged across the unused RM factors, by ticking the corresponding box? This can also have a big impact on the result. It's a bit puzzling though why also the Bayesian RM ANOVA is giving different results. Would it be possible for you to share the jasp file or data set, so that I can look into this issue? You can also send it by email to j dot b dot vandoorn at uva dot nl

Kind regards, Johnny

JohnnyDoorn commented 3 years ago

Hi @Angalanse ,

Thanks for the data set. The discrepancy of the results comes from the fact that JASP applies a correction to the intervals (see also the help file for the RM ANOVA):

In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure. The paper in question can be found here.

below is R code that reproduces both versions of the results (SPSS first for the uncorrected intervals, and then the JASP results with the corrected results).

Kind regards, Johnny

dat <- read.csv2("~/Downloads/ForVanDoorn.csv", dec = ",")

# convert wide data to long data
longDat <- reshape2::melt(dat, measure.vars = 3:11)

# compute anova and marginal means with afex and emmeans UNCORRECTED
res <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")
mMeans <- emmeans::emmeans(res, specs = c("variable", "Sex"))

# compute pp means to apply Morey correction
idMeans <- tapply(longDat$value, longDat$FPID, mean)
longDat$value <- longDat$value - idMeans[longDat$FPID]

# compute anova and marginal means with afex and emmeans CORRECTED
resCor <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")
mMeansCor <- emmeans::emmeans(resCor, specs = c("variable", "Sex"))

# plot marginal means
plot(mMeans)
plot(mMeansCor)
Angalanse commented 3 years ago

Thanks Johnny! That’s interesting!

L

Från: JohnnyDoorn notifications@github.com Skickat: den 2 mars 2021 10:19 Till: jasp-stats/jasp-issues jasp-issues@noreply.github.com Kopia: Linus Andersson linus.andersson@umu.se; Mention mention@noreply.github.com Ämne: Re: [jasp-stats/jasp-issues] Incorrect CI bars for repeated measures ANOVA (standard and Bayesian) (#1187)

Hi @Angalansehttps://github.com/Angalanse ,

Thanks for the data set. The discrepancy of the results comes from the fact that JASP applies a correction to the intervals (see also the help file for the RM ANOVA):

In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure. The paper in question can be found herehttp://pcl.missouri.edu/sites/default/files/morey.2008.pdf.

below is R code that reproduces both versions of the results (SPSS first for the uncorrected intervals, and then the JASP results with the corrected results).

Kind regards, Johnny

dat <- read.csv2("~/Downloads/ForVanDoorn.csv", dec = ",")

convert wide data to long data

longDat <- reshape2::melt(dat, measure.vars = 3:11)

compute anova and marginal means with afex and emmeans UNCORRECTED

res <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")

mMeans <- emmeans::emmeans(res, specs = c("variable", "Sex"))

compute pp means to apply Morey correction

idMeans <- tapply(longDat$value, longDat$FPID, mean)

longDat$value <- longDat$value - idMeans[longDat$FPID]

compute anova and marginal means with afex and emmeans CORRECTED

resCor <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")

mMeansCor <- emmeans::emmeans(resCor, specs = c("variable", "Sex"))

plot marginal means

plot(mMeans)

plot(mMeansCor)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jasp-stats/jasp-issues/issues/1187#issuecomment-788755773, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATBIGIVJ5WETBOJGQM7YZITTBSUP5ANCNFSM4YMXACFA.

Angalanse commented 3 years ago

Just a follow-up Johnny,

One of the arguments in the paper you linked to was that the new confidence intervals are more aligned with the results of the (significant) t-test in the paper. The pedagogical problem I see with the RM ANOVA example I provided is that readers who are used to glancing at the figure to get an overview of the results would probably expect an immense difference between the two factors (men / women). However, there is no supported / significant difference.

L

Från: JohnnyDoorn notifications@github.com Skickat: den 2 mars 2021 10:19 Till: jasp-stats/jasp-issues jasp-issues@noreply.github.com Kopia: Linus Andersson linus.andersson@umu.se; Mention mention@noreply.github.com Ämne: Re: [jasp-stats/jasp-issues] Incorrect CI bars for repeated measures ANOVA (standard and Bayesian) (#1187)

Hi @Angalansehttps://github.com/Angalanse ,

Thanks for the data set. The discrepancy of the results comes from the fact that JASP applies a correction to the intervals (see also the help file for the RM ANOVA):

In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure. The paper in question can be found herehttp://pcl.missouri.edu/sites/default/files/morey.2008.pdf.

below is R code that reproduces both versions of the results (SPSS first for the uncorrected intervals, and then the JASP results with the corrected results).

Kind regards, Johnny

dat <- read.csv2("~/Downloads/ForVanDoorn.csv", dec = ",")

convert wide data to long data

longDat <- reshape2::melt(dat, measure.vars = 3:11)

compute anova and marginal means with afex and emmeans UNCORRECTED

res <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")

mMeans <- emmeans::emmeans(res, specs = c("variable", "Sex"))

compute pp means to apply Morey correction

idMeans <- tapply(longDat$value, longDat$FPID, mean)

longDat$value <- longDat$value - idMeans[longDat$FPID]

compute anova and marginal means with afex and emmeans CORRECTED

resCor <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")

mMeansCor <- emmeans::emmeans(resCor, specs = c("variable", "Sex"))

plot marginal means

plot(mMeans)

plot(mMeansCor)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jasp-stats/jasp-issues/issues/1187#issuecomment-788755773, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATBIGIVJ5WETBOJGQM7YZITTBSUP5ANCNFSM4YMXACFA.

AugustOlsson commented 2 years ago

@JohnnyDoorn I have a related question. I have noticed in later editions of JASP that 95% Credible-intervals of RM-ANOVAS are much smaller than the 95% credible interval given in the Descriptives tables. As I understand the above discussion this is intended for frequentist confidence intervals, but not for bayesian credible intervals. Is it then a bug? I am currently using old intervals for my paper that I plotted with an older version of JASP where there was no discrepancy between the 95% credible intervlas from the descriptives table and the 95% credible intervals from the descriptive plots. But I would like to know if the smaller intervals that are currently plotted are actually "more" correct.

PS. Note if I should start a new issue on this, but it seems highly related.

EDIT: Basically the Credible intervals look identical to the Confidence intervals (which I think is by design given that a uniform prior is used for the decriptive plots) - thus I guess the same correction is used for the Bayesian Credible intervals. Is that resonable?

Best, August

tomtomme commented 7 months ago

@AugustOlsson Can you test this with the current JASP version 0.18.3 and share your .jasp file? Maybe then someone (@JohnnyDoorn ?) can answer this, if still relevant. Thx

AugustOlsson commented 7 months ago

@tomtomme

I'll test it during next week and get back to you!

tomtomme commented 6 months ago

@AugustOlsson I have tested this now with 0.19 beta. Both RM ANOVAs show identical CIs in tables and in plots. So what you witnessed with the discrepancy between table and plot regarding the credible interval must have been a bug - thats now corrected. Both seem to apply the Morey correction - but the Bayesian help file does not mention this. I will add this to the needed list of improvements for the help files here: #2529

@Angalanse Closing for now, since #1227 tracks if the correction could be optional to avoid the pedagocical problem you mentioned.

AugustOlsson commented 4 months ago

Thanks for the update @tomtomme. I would really really like the option to get CIs without Morey correction though. If it was possible to include a choice to add/not add Morey correction that would be greatly apprechiated.

Best, August

tomtomme commented 4 months ago

Yep. This is still on the todo list at #1227

tomtomme commented 4 months ago

@AugustOlsson This is likely getting done for upcoming 0.19 version of jasp :)