Filtering data changes the output (still)

[K1NDRED]

JASP Issue dataset.zip

• JASP version: 0.11.1

• OS name and version: Windows 10 Home

• Analysis: Bayesian RM ANOVA

• Bug description: I have 2 training groups who have a pre- and a post-training measure on a variable. Each training group contains young adults and older adults. When I filter the data to look at one training group using the filter icon button, I get one Bayes output. When I filter the data by clicking on the variable name and then X-ing out the unwanted training group, I get a different output. To make matters worse, even if I choose to filter my training group using one method, every time I do it, I get a different output. Despite closing everything to make sure there is no analysis running at the same time as I filter.

• Expected behaviour: The same output for both filtering methods. The same output for the same filtering method.

Steps to reproduce:

1. Go to the data window
2. Click on ''Training_group" column
3. Filter out the NB group by X-ing it (see pic 1)
4. Run a Bayesian RM Anova (see pic 2 & 2a), note the values of the output
5. Delete the Bayesian RM Anova from the output
6. Go back to the data window & Click on ''Training_group" column
7. Click the X to remove the filter so that both training groups are available
8. Filter out the NB group again by X-ing it
9. Run the Bayesian RM Anova again (see pic 2 & 2a), note the values of the output are different.
10. Refreshing the results gives different outputs constantly

[boutinb]

@JohnnyDoorn Could you look at this issue?

[JohnnyDoorn]

Hi @K1NDRED ,

Could you give some screenshots of the different output? How big are these changes? The Bayesian ANOVA's use MCMC-sampling, so it is expected that there are small differences between subsequent runs, but these differences typically should not lead to qualitatively different conclusions.

Another solution to your problem could be to try this with the latest version of JASP. I cannot seem to reproduce this issue.

Kind regards, Johnny

[K1NDRED]

Hi @K1NDRED ,

Could you give some screenshots of the different output? How big are these changes? The Bayesian ANOVA's use MCMC-sampling, so it is expected that there are small differences between subsequent runs, but these differences typically should not lead to qualitatively different conclusions.

Another solution to your problem could be to try this with the latest version of JASP. I cannot seem to reproduce this issue.

Kind regards, Johnny

Hello,

Thank you for your response. Upon further testing, the differences aren't large enough to change the qualitative conclusions.

This might not be the place but I'm not sure I understand why using the MCMC-sampling causes these variations.

Cheers

[JohnnyDoorn]

Hi @K1NDRED ,

MCMC-sampling is a computational technique that is at the core of many Bayesian analyses. This technique uses random number sampling in order to approximate the posterior distribution, which is what Bayesian analyses are based on. Because of the sampling, every time you run an analysis the numbers will vary slightly (but should not lead to qualitatively different conclusions).

I will close this issue now, please reopen if anything is still unclear!

Kind regards Johnny