Set priors on the absolute (raw) and not not on the effect size scale

[alessio-toraldo]

Description

I just mentioned the problem to Eric (Wagenmakers) in a zoom meeting in Trento (CimeC), and he also recalled that the same point was made by Zoltan Dienes. Priors (and posteriors) should be set, and shown, in their original scale and not in the effect size scale, which makes them little intelligible. This is especially true for students, but also for analysts.

Purpose

To make results much clearer to students (and scholars!)

Use-case

No response

Is your feature request related to a problem?

No response

Is your feature request related to a JASP module?

Unrelated

Describe the solution you would like

Set the default scale of priors and posteriors to the original scale, and leave the effect-size scale as a secondary choice.

Describe alternatives that you have considered

No response

Additional context

No response

[EJWagenmakers]

But we have this, which implements Dienes' procedures from an R package: https://jasp-stats.org/2022/01/18/new-in-jasp-0-16-zoltan-dienes-general-bayesian-tests/ Is this not what is needed?

[alessio-toraldo]

Not exactly. Zoltan's procedure requires that one extracts "by hand" the mean and SE of an estimate, and plugs them in the calculator. What I was wondering is: why can't we skip those passages, and have that procedure (implicitly) built-in in all JASP commands? All that is required is just changing the scale of the visualization of priors and posteriors, as well as the scale of the prior setting, to the original scale (it is now in the delta scale, effect size). Or, even better, allow the user to choose between the two scales, standardized (delta) or raw. I also suspect that Dienes's procedure is a shortcut that does not do exactly the same computation as a full model comparison, when there are more than one parameter in the model. Dienes's procedure should work fine when the model has one parameter. Say, a one-sample t-test, so the two models to be compared have Theta=0 and Theta=some non-zero prior distribution. What I'm not sure about is whether Dienes's calculator works also when the models are more complex, say, when one is testing the slope of a linear regression. Here the Bayesian comparison is between a model with zero slope and some intercept (with some non-zero prior), and a model with two non-zero priors, one for slope and one for intercept. I believe (I might be wrong here!) that using BF to compare the two models integrates the likelihood at numerator over both the prior of the intercept and the prior of the slope, and integrates the likelihood at denominator over the prior of the intercept (also, one might consider a prior on the correlation between slope and intercept, but that's another story). I am not sure Dienes's procedure, used with the slope's estimate's M and SE as input, gives the same BF as the one given by JASP from the linear regression. I suspect (again, I might well be wrong - need to check) that the BF might depend on the prior for the intercept, which Dienes's procedure ignores.

[FBartos]

Hi Alessio,

Just a quick comment on the Dienes procedure—you can actually show that it holds for any nested model comparison (it is essentially equivalent to Savage-Dickey density ratio, see e.g. https://doi.org/10.1002/sta4.600 for explanation). In case of linear regression with the usual JZS-priors, you will get slightly different results because the prior on the effect size is not independent of the prior on sigma. If you compare to a different set-up, you could use the test for beta coefficients as well. You can also check Johnson and colleagues Bayes factor functions (https://doi.org/10.1073/pnas.2217331120) that use the same principle.

Regarding the implementation. This could indeed be added to each analysis where the estimate would be followed by a BF based on the estimate and standard error. However, I would argue that it would be a non-standard feature that would confuse many practitioners and clutter the interface. Currently, JASP offers the Summary Statistics module that allows for all the analyses—you just need to copy the estimate and the standard error. The 'General Bayesian Tests' and Bayesian Z-Test' analyses allow you to perform these kinds of analyses and offer additional visualization options that wouldn't be possible to pack into each analysis.

Cheers, Frantisek

[alessio-toraldo]

Hi Frantisek tx for your response and precious refs, I will go through them. As to the implementation, I see the advantages of the new modules, 'General Bayesian Tests' and Bayesian Z-Test', and I also agree that keeping them separate from other analyses is a good choice. However, what I am suggesting is way simpler than what you seemed to impy, and would not clutter the interface at all: it would include just a single additional checkbox allowing the user to choose whether the scale for priors/posteriors should be the raw or the delta one. It would not confuse the practitioners, quite the contrary. It's the delta scale that is puzzling, because the standardization factor is often not at all obvious. This is especially true for complex designs and higher-order terms. Most practitioners do not know what the scaling factor is for, e.g., a triple interaction, and also consider that the scaling factors are different for different terms, which currently makes the whole procedure cumbersome. Else, one single click would instantly provide them with posteriors on the intelligibile scale, say, mm for blood pressure. Yet another advantage is that users would have better control over the priors - currently, they select one of the default prior distributions without real knowledge of what they are doing. Having the original scale would make them choose sensible priors. I believe this is actually the standard choice for most softwares...? E.g. R packages use raw-scale priors as default. But perhaps I am missing something of what you implied here? Alessio

[tomtomme]

@alessio-toraldo Duplicate? https://github.com/jasp-stats/jasp-issues/issues/1664

[alessio-toraldo]

Yes, that is the same problem. I forgot that the point had emerged in class then (2022) and that I had already signalled it. So erase the first. Best Alessio

Il mer 14 feb 2024, 13:12 Thomas Langkamp @.***> ha scritto:

@EJWagenmakers https://github.com/EJWagenmakers From what I read here: #1664 https://github.com/jasp-stats/jasp-issues/issues/1664 you yourself requested this two years ago, so I would close this as a duplicate

@alessio-toraldo https://github.com/alessio-toraldo What do you think?

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/2559#issuecomment-1943649843, or unsubscribe https://github.com/notifications/unsubscribe-auth/BFZHRIVYK5BNR4I7DEYGXS3YTSSZFAVCNFSM6AAAAABCVHAINCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGY2DSOBUGM . You are receiving this because you were mentioned.Message ID: @.***>