Different JASP different output - same data

[DrMarkGS]

JASP Version

0.18.3

Commit ID

No response

JASP Module

T-Tests

What analysis are you seeing the problem on?

Paired samples

What OS are you seeing the problem on?

Windows 11

Bug Description

Different values for the same data in JASP 0.16 and 0.18.3 I am updating my guide to reflect changes since 0.16. I used the same data in both for a Wilcoxon's signed rank test. The output is almost the same except that the rank biserial correlation is completely different. 0.16 rB = 0.48 0.18.3 rB = 0.982

What has changed? Mark

Expected Behaviour

see above

Steps to Reproduce

1. 2. 3. ... see above

Log (if any)

No response

Final Checklist

• [X] I have included a screenshot showcasing the issue, if possible.
• [ ] I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
• [ ] I have accurately described the bug, and steps to reproduce it.

[DrMarkGS]

As a follow on the descriptive data is also different. Median values are the same however the MAD values are different. 0.16 - mdn 22, MAD 7.41 0.18.3 - mdn 22, MAD 5

[DrMarkGS]

.....and another. One sample t-test - Shapiro-Wilk 0.16 - W=0.969 p .507 0.18.3 W = 0.983 p .954

[JohnnyDoorn]

Hi @DrMarkGS,

Can you include the data? Then I can verify the results in R. I found this commit that fixed the computation of the rank-biserial correlation: https://github.com/jasp-stats/jaspTTests/commit/492ab0fed7e25be460cec2c5c67439779e7bf72a Although that only goes for the one-sample t-test and not the paired. The code for the shapiro test and rank-biseral test don't seem to be changed since April 2020, and 0.16 is from 2021.

edit: I found the data from your OSF repo. For the rank-biserial correlation, the old computation seemed to not have accounted for ties in the data when standardizing the observed W.

for the MAD, this now seems to be ran as mad(x, constant = 1), instead of only mad(x). To get the original value, you now need to use "MAD robust". From the helpfile: "MAD robust: Median absolute deviation of the data points, adjusted by a factor for asymptotically normal consistency." which seems to be the case when you do not use constant = 1 (so in line with what we do in JASP).

For the shapiro test I can only say that the current value is corresponding to R's shapiro.test(), after checking with your data set. I'm not sure why the previous value is different, since as far as I can see that one also used the same R function.

Cheers Johnny

[tomtomme]

@DrMarkGS @JohnnyDoorn To clarify: This leaves to explain the difference in Wilcoxon signed rank for pairs? Or is this also now corrected for ties? And for shapiro.test - maybe there was some change in the underlying R-package?

--

Related: https://github.com/jasp-stats/jasp-issues/issues/2220

[JohnnyDoorn]

The signed rank stuff was about the rank-biserial, so that is resolved now. The Shapiro test is still a bit mysterious, since we do have a unit test in place, but there the value did not change in the last 4 years according to git-blame.

[tomtomme]

Ok @DrMarkGS Do you think we can close this one then?

[DrMarkGS]

Thanks Thomas I guess so

Mark Goss-Sampson Professor of Human Movement Faculty of Education, Health & Human Sciences School of Human Sciences University of Greenwich

Telephone: +442083317898 | Email: @.**@.>

From: Thomas Langkamp @.> Sent: Thursday, March 14, 2024 7:20 PM To: jasp-stats/jasp-issues @.> Cc: Mark Goss-Sampson @.>; Mention @.> Subject: [EXTERNAL] Re: [jasp-stats/jasp-issues] Different JASP different output - same data (Issue #2636)

This message originated from outside the University. Treat links and attachments with caution.

---

Ok @DrMarkGShttps://github.com/DrMarkGS Do you think we can close this one then?

— Reply to this email directly, view it on GitHubhttps://github.com/jasp-stats/jasp-issues/issues/2636#issuecomment-1998232877, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJJSHAAC3HF2XBOATTPT3D3YYHZ5ZAVCNFSM6AAAAABEUJP36SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJYGIZTEOBXG4. You are receiving this because you were mentioned.Message ID: @.***>

University of Greenwich, a charity and company limited by guarantee, registered in England (reg no. 986729). Registered Office: Old Royal Naval College, Park Row, Greenwich SE10 9LS.

[DrMarkGS]

Hi folks I see that you have also tinkered with the Cohen’s d calculations in RMANOVA

0.16 @.***

0.18.3 @.***

Mark Goss-Sampson Professor of Human Movement Faculty of Education, Health & Human Sciences School of Human Sciences University of Greenwich Telephone: +442083317898 | Email: @.**@.>

From: Johnny van Doorn @.> Sent: Thursday, March 14, 2024 4:38 PM To: jasp-stats/jasp-issues @.> Cc: Mark Goss-Sampson @.>; Mention @.> Subject: [EXTERNAL] Re: [jasp-stats/jasp-issues] Different JASP different output - same data (Issue #2636)

This message originated from outside the University. Treat links and attachments with caution.

---

The signed rank stuff was about the rank-biserial, so that is resolved now. The Shapiro test is still a bit mysterious, since we do have a unit test in place, but there the value did not change in the last 4 years according to git-blame.

— Reply to this email directly, view it on GitHubhttps://github.com/jasp-stats/jasp-issues/issues/2636#issuecomment-1997871184, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJJSHAFFPK2EUXY3CTXCLG3YYHG45AVCNFSM6AAAAABEUJP36SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJXHA3TCMJYGQ. You are receiving this because you were mentioned.Message ID: @.***>

University of Greenwich, a charity and company limited by guarantee, registered in England (reg no. 986729). Registered Office: Old Royal Naval College, Park Row, Greenwich SE10 9LS.

[EJWagenmakers]

I am a little puzzeled how this could have happened, since we have our unit tests. Any thoughts?

[JohnnyDoorn]

Hi @DrMarkGS ,

Yes, this is now handled by the emmeans package - the relevant commit is here: https://github.com/jasp-stats/jaspAnova/commit/1305b7ac88c252b5abcd6d22492e2c9962b03127

@EJWagenmakers when we tweak such a calculation we also update the unit test. The only thing i'm puzzled by is the shapiro wilk test in t-test (which does have a unit test as I mentioned above).

Cheers, Johnny