search

jasp-stats / jasp-issues

This repository is solely meant for reporting of bugs, feature requests and other issues in JASP.
58 stars 29 forks source link

[Bug]: Different Friedman test results between two versions of JASP #1710

Closed clemergo closed 2 years ago

clemergo commented 2 years ago

JASP Version

0.16.2

Commit ID

No response

JASP Module

jaspAnova

What analysis are you seeing the problem on?

Friedman test (non-parametric repeated measures ANOVA)

What OS are you seeing the problem on?

Win10 (v0.16.2) and Win7 (v0.11)

Bug Description

I have two computers one running win7 and one running win10. Each run different versions of JASP (respectively 0.11.0 and 0.16.2).

When running a Friedman test (non-parametric repeated measures ANOVA) on the same data, I get very different results. What explains this difference and which is the correct one? The difference between the two could be explained by this fix (link), but in my case Kendall's W is not the only value affected.

0.11.0 result 0 11 0friedman 0.16.2 result 0 16 2friedman

Expected Behaviour

Having the same results.

Steps to Reproduce

  1. Select "Repeated measure Anova"
  2. In the "Model" panel move the "RM Factor 1" factor to the "Model Terms" block
  3. In the "Nonparametrics" panel move the "RM Factor 1" factor to the "RM factor pannel"
  4. To this for the two versions of Jasp ...

The data can be found here : https://filesender.renater.fr/?s=download&token=f3c72452-a1c6-486b-9c3f-3dffc7f5d964

Log (if any)

No response

Final Checklist

juliuspfadt commented 2 years ago

Hi @clemergo, thanks for reporting this. @JohnnyDoorn might have an idea why this happens.

JohnnyDoorn commented 2 years ago

Hi @clemergo ,

The result from 0.11 was when the Friedman test was just implemented. Unfortunately I discovered a small bug shortly afterwards, and fixed the computation (updated to use code from the R-package PMCMRplus). However, I now see that we are too permissive in allowing the Friedman test, which has a strict requirement for the design to be unreplicated. When you have a 2x2 within subjects design, you have inherent replicated observations (e.g., an observation for participant 1, in condition 1A, for both conditions 2A and 2B - where 1 and 2 refer to the RM factors, and A and B to their levels). I will add an additional check, to prevent potentially misleading use of the Friedman test. Apologies that I do not have a more satisfying answer for you - in the end the results of 0.16.2 are more accurate than 0.11. I will close this issue now, please reopen if you have any more questions.

Kind regards, Johnny

clemergo commented 2 years ago

Hi @JohnnyDoorn

Thank you very much for your very helpful (and satisfying) reply! I'm sorry I forgot to reply to you in a timely manner...

Bests, Clement