[Bug]: Negative Sum of Squares in Repeated Measures ANOVA (classic)

chris-olivers commented 3 weeks ago

JASP Version

0.18.3

Commit ID

No response

JASP Module

ANOVA

What analysis are you seeing the problem on?

Classic repeated measures anova

What OS are you seeing the problem on?

Windows 11

Bug Description

I get negative F-values on a main effect. This results from negative Sum of Squares. Negative SS should not be possible, right?

Expected Behaviour

Positive sum of squares

Steps to Reproduce

[1.] See attached JASP file 2. 3. ... fix closest to.zip

Log (if any)

No response

Final Checklist

[ ] I have included a screenshot showcasing the issue, if possible.
[X] I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
[ ] I have accurately described the bug, and steps to reproduce it.

JohnnyDoorn commented 3 weeks ago

Hi @chris-olivers ,

I cannot replicate this on my machine (MacOS with 0.18.3) - refreshing your JASP file gives me positive values everywhere. Maybe this is Windows specific, when very small numbers are involved. Could you see if this is still an issue with the latest development verison, available here?

TarandeepKang commented 3 weeks ago

Hi Johnny, Obviously, I'm not Chris but I was intrigued by this bug in any case. So I downloaded the nightly from above. When I tried to download the attached just file or anything from the library I get this error message. Like him, I'm also on Windows 11.

Best,

Tarandeep

JohnnyDoorn commented 3 weeks ago

Hi @TarandeepKang Haha yes it is a very intriguing issue! Which is why I was a bit sad I couldn't replicate it.. Does this happen when you load any analysis? In that case maybe the windows nightlies are broken and we might need to wait for the more official release.. are you able to reproduce this issue in 0.18.3 on windows?

TarandeepKang commented 3 weeks ago

I could replicate the original bug with the negative SS with 0.18.3 on my Windows 11 machine. As for this thing that popped up on my screen that I showed you above, never seen it before. Only when I downloaded the nightly that you linked to above.

chris-olivers commented 2 weeks ago

Thanks both for looking into it. “Good” that Tarandeep can replicate it. If you don’t mind I prefer onto install the beta msi, as I’’m worried it may clash with my current install.

I did enter the same data set in SPSS and get a totally different outcome:

Here JASP result: Within Subjects Effects Cases Sum of Squares df Mean Square F p η² pre_post_int 2.041×10-26 1 2.041×10-26 -4.178×10-12 1.000 1.216×10-30 Residuals -1.563×10-13 32 -4.885×10-15

fix_to_dist_targ 11680.446 1 11680.446 140.414 < .001 0.696 Residuals 2661.948 32 83.186

pre_post_int ✻ fix_to_dist_targ 705.567 1 705.567 12.975 0.001 0.042 Residuals 1740.185 32 54.381

Note. Type III Sum of Squares

Here SPSS result for exact same data: @.***

Different F values, different SS (no small numbers at all....)

Intriguingly though, JAMOVI leads to the exact same error:

Within Subjects Effects

Sum of Squares df Mean Square F p RM Factor 1 2.04e-26 1 2.04e-26 -4.18e−12 1.000 Residual -1.56e−13 32 -4.88e−15

RM Factor 2 11680 1 11680.4 140.4 < .001 Residual 2662 32 83.2

RM Factor 1 ✻ RM Factor 2 706 1 705.6 13.0 0.001 Residual 1740 32 54.4

Note. Type 3 Sums of Squares

Hope this helps.

From: TarandeepKang @.> Sent: 05 June 2024 18:14 To: jasp-stats/jasp-issues @.> Cc: Olivers, C.N.L. (Christian) @.>; Mention @.> Subject: Re: [jasp-stats/jasp-issues] [Bug]: Negative Sum of Squares in Repeated Measures ANOVA (classic) (Issue #2766)

I could replicate the original bug with the negative SS with 0.18.3 on my Windows 11 machine. As for this thing that popped up on my screen that I showed you above, never seen it before. Only when I downloaded the nightly that you linked to above.

— Reply to this email directly, view it on GitHubhttps://github.com/jasp-stats/jasp-issues/issues/2766#issuecomment-2150455083, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCHQQAYCD52ZC62QGRKBX5DZF42LLAVCNFSM6AAAAABI2XMYUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJQGQ2TKMBYGM. You are receiving this because you were mentioned.Message ID: @.***>

JohnnyDoorn commented 2 weeks ago

Hi @chris-olivers

Could you include a screenshot from the SPSS output (or ideally, SPSS syntax?) - then I can verify that both analyses are ran the same way. We have checked our RM ANOVA fairly extensively against other software, so I'm curious how such a large mismatch can occur.

edit: Here is R-code to recreate the analysis:

library(tidyr)
dat <- read.csv("~/Downloads/negss.csv")

# Convert to long format
long_dat <- dat %>%
  pivot_longer(
    cols = starts_with("pre") | starts_with("post"),
    names_to = c("prePostInt", "fixDistTarg"),
    names_sep = "_",
    values_to = "dv"
  )

# Adjust the levels
long_dat$prePostInt <- factor(long_dat$prePostInt, levels = c("pre", "post"))
long_dat$fixDistTarg <- factor(long_dat$fixDistTarg, levels = c("dist", "target"))

afex::aov_ez(id = "sub_nr", dv = "dv", data = long_dat, within = c("prePostInt", "fixDistTarg"))

gives result:

Anova Table (Type 3 tests)

Response: dv
                  Effect    df   MSE          F   ges p.value
1             prePostInt 1, 32  0.00       0.00 <.001   >.999
2            fixDistTarg 1, 32 83.19 140.41 ***  .726   <.001
3 prePostInt:fixDistTarg 1, 32 54.38   12.97 **  .138    .001
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1

Kind regards, Johnny

chris-olivers commented 2 weeks ago

Hi, sorry, made a mistake in the SPSS analysis: Decimal points were converted wrongly when copy-pasting the data. When done right SPSS spits out:

Tests of Within-Subjects Effects Measure: MEASURE_1 Source Type III Sum of Squares df Mean Square F Sig. factor1 Sphericity Assumed ,000 1 ,000 . . Greenhouse-Geisser ,000 . . . . Huynh-Feldt ,000 . . . . Lower-bound ,000 1,000 ,000 . . Error(factor1) Sphericity Assumed ,000 32 ,000

Greenhouse-Geisser ,000 . .

Huynh-Feldt ,000 . .

Lower-bound ,000 32,000 ,000

factor2 Sphericity Assumed 116804461,364 1 116804461,364 140,414 <,001 Greenhouse-Geisser 116804461,364 1,000 116804461,364 140,414 <,001 Huynh-Feldt 116804461,364 1,000 116804461,364 140,414 <,001 Lower-bound 116804461,364 1,000 116804461,364 140,414 <,001 Error(factor2) Sphericity Assumed 26619481,636 32 831858,801

Greenhouse-Geisser 26619481,636 32,000 831858,801

Huynh-Feldt 26619481,636 32,000 831858,801

Lower-bound 26619481,636 32,000 831858,801

factor1 factor2 Sphericity Assumed 7055669,121 1 7055669,121 12,975 ,001 Greenhouse-Geisser 7055669,121 1,000 7055669,121 12,975 ,001 Huynh-Feldt 7055669,121 1,000 7055669,121 12,975 ,001 Lower-bound 7055669,121 1,000 7055669,121 12,975 ,001 Error(factor1factor2) Sphericity Assumed 17401845,879 32 543807,684

Greenhouse-Geisser 17401845,879 32,000 543807,684

Huynh-Feldt 17401845,879 32,000 543807,684

Lower-bound 17401845,879 32,000 543807,684

So it cannot compute factor1, the factor that gives the negative value in JASP.

I am also starting to see why, which would mean it’s not a bug.... The way the student dis the analysis means that the data in the columns are not independent, i.e. they are complementary per two columns. Specifically they are percentages that add up to 100..

What JASP then probably latched onto were very tiny rounding differences.

I apologize, I did not notice this – my bad.

Chris

From: Johnny van Doorn @.> Sent: 05 June 2024 21:49 To: jasp-stats/jasp-issues @.> Cc: Olivers, C.N.L. (Christian) @.>; Mention @.> Subject: Re: [jasp-stats/jasp-issues] [Bug]: Negative Sum of Squares in Repeated Measures ANOVA (classic) (Issue #2766)

Hi @chris-olivershttps://github.com/chris-olivers

Could you include a screenshot from the SPSS output (or ideally, SPSS syntax?) - then I can verify that both analyses are ran the same way. We have checked our RM ANOVA fairly extensively against other software, so I'm curious how such a large mismatch can occur.

Kind regards, Johnny

— Reply to this email directly, view it on GitHubhttps://github.com/jasp-stats/jasp-issues/issues/2766#issuecomment-2150834620, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCHQQAYLUID73VXEXUDYULTZF5TRNAVCNFSM6AAAAABI2XMYUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJQHAZTINRSGA. You are receiving this because you were mentioned.Message ID: @.**@.>>

jasp-stats / jasp-issues