BF of one-sample wilcoxon seems random

[YvonneOberholzer]

• JASP version: 0.12.2
• OS name and version: Windows 10 Home
• Analysis: Bayesian One-Sample Wilcoxon Test
• Bug description: The analysis yields different Bayes Factors every time I run it (same data). So far, the BFs have ranged from something around 4'000 to 9'000'000.
• Expected behaviour: Somehwat consistent BFs.

Steps to reproduce:

1. Go to 'T-Tests'
2. Click on 'Bayesian One-Sample T-Test'
3. Select '< Test value, BF10, Wilcoxon signed-rank, Exclude cases per dependent variable, Additional Statistics: Descriptives'
4. Wait for iterations
5. Repeat several times
6. See error (inconsistency)

conf_diff_2_inconsistent_BFs.zip

[JohnnyDoorn]

Hi @YvonneOberholzer,

The algorithm used for the Bayesian Wilcoxon tests is heavily dependent on MCMC-sampling because of the data augmentation that is needed in the analysis. Espcially if there is a large effect size, there will be quite some fluctuation of the Bayes factor, but the qualitative conclusions should not differ from each other. Looking at your file, it seems that all Bayes factors are larger than 10,000, which is how I would report it (i.e., " BF_+0 > 10,000). The new JASP version (0.13) will have an additional step that will stabilize the results somewhat (increasing the number of samples will also help a little), but these fluctuations are inherent to the analysis, unfortunately.

Kind regards Johnny

[YvonneOberholzer]

Hi @JohnnyDoorn ,

thank you for your reply and explanation! The BF-0 is not always above 10'000 though. In one of my first analyses, it was around 4'000 and in one of a more recent analysis, it was 1'463 (see attachment). So I am not really confident in reporting it as above 10'000 or even as above 1000. If it is heavily dependent on the MCMC-sampling, would it help to increase the number of samples? I tried 5'000 in the first file, but it did not seem to change much.

Best regards, Yvonne

conf_diff_2_inconsistent_BFs2.zip