BFs to Test Interaction Term Don't Tie for Equivalent Models

[mattstern2]

Basic issue: I code a factor for the interaction in a 2x2 ANOVA and compare its BF (relative to intercept only model) to the BFs for the interaction term that's automatically coded by BayesFactor package. The BFs don't tie.

Details: I have two factors that have either a HIGH or LOW value. To manually code the interaction, I use the following:

F1=HIGH & F2=HIGH => Int=HIGH F1=HIGH & F2=LOW => Int=LOW F1=LOW & F2=HIGH => Int=LOW F1=LOW & F2=LOW => Int=HIGH

I then run code for models (see attached code) that include the "Int" variable. Should not the "F1 + F2 + Int" model have the same BF as the "F1 + F2 + F1:F2" model (both relative the the intercept only model)? The BFs don't tie.

To investigate one potential source of the problem, I looked at the model matrices for the two models that should be equivalent. They code the variables differently, as expected. In particular, F1:F2 is coded as +/-0.5 while Int is coded as +/-Sqrt(2)/2, depending on the level of the interaction term.

At the same time, the BF for a model with only the interaction (versus intercept only model) is the same for "F1:F2" and "Int", despite the different parameterizations.

I am very confused.

Update: I removed a comment about equation 11 in Rouder, Morey, Speckman, and Province (2012) as the referenced equation pertained to one-way ANOVA and my example is a two-way ANOVA.

BayesFactor Issue (Interaction).txt

[mattstern2]

Update: I found another potential bug: ttestBF does not tie to anovaBF when the scales are set to the same value. Combine the following code with the above "BayesFactor Issue (Interaction).txt)" file:

anovaBF(DV~F1, data = testdata, rscaleFixed = 1) ttestBF(formula = DV ~ F1, data = testdata, rscale = 1)

You get the same BFs if you use you set rscale = 2*rscaleFixed/sqrt(2).

[richarddmorey]

Re: the second question, this is mentioned under the "Note" in the help for ttestBF. The priors were parametrised slightly differently in different papers and are thus different by a multiplicative constant, so using the same numerical value for both will result in two different Bayes factors. That is why the prior label "medium" was introduced: to make sure things are equal across calls.

To get the same result, you need to multiply the prior scale by sqrt(2), as in the following code:

d = data.frame(x = factor(rep(c(0,1),each=5)), y = rnorm(10))
rscaleA = 1
rscaleT = sqrt(2) * rscaleA
BayesFactor::ttestBF(formula = y ~ x, data = d, rscale = rscaleT)
BayesFactor::anovaBF(y ~ x, rscaleFixed = rscaleA, data = d)

It is not a bug.

[mattstern2]

Thanks, Richard - sorry for not being more thorough while reading the documentation. For question 1, there are clearly different parameterizations and the effects sizes are not doubly standardized, so BFs will be different (I see all this explained in in Rouder, Morey, Speckman, and Province 2012).

[mattstern2]

I closed this comment, but I have a simple question that I can't seem to figure out.

Is it the case that the interaction term is on a different scale than the main effects? Reading RMSP 2013, I see that the interaction is coded as the product of the main effect columns (I can check the design matrix with model.matrix() and see that this is the case). For a 2x2 fixed effects model, the interaction is +/-0.5 while the main effects are +/-sqrt(2)/2. So the IVs are on different scales and, since no normalization of effects is applied for ANOVA, for a given rscaleFixed, the scale on the main effects and interaction should be different.

But there doesn't appear to be a difference in scale when only the interaction term is included in the model. I think that this is because the code defaults to a t test when nFactors = 1 & nLvls = 2 (as seen in methods-BFlinearModel-compare.R code). A conversion for the scale is applied per your comment, but the parameterization for the interaction is different from the main effects. This means that an interaction with an equivalent t value should have a different BF, I believe. Unless, the scale intentionally adjusted to account for the different parameterization of the interaction.

So, my concern is: is the scale for the interaction term adjusted when the nWayFormula is applied? If not, it seems the scale on the interaction term is different when the interaction only model is compared to the intercept only model versus when the full model is compared to the main effects only model.

I did my best to read through the documentation and code to get an answer, but I couldn't figure it out.

[mattstern2]

I've carefully reviewed all relevant papers and documentation. I think I found a bug, and one undesirable feature.

I compare the output from anovaBF to a generalTestBF with the appropriate scale adjustment. Three issues:

The BFs for the main effects only models do not tie, despite the fact that the single factor models do.

The BFs for the interaction only model DO tie, despite the interaction having a different parameterization (with the ANOVA approach its set to +/-0.5, as can be verified running model.matrix()). I believe that this is because the anovaBF code defaults to a ttest when there's one factor with two levels. The scale is adjusted, but not in a way that accounts for the fact that the interaction has a different coding (+/-0.5 as opposed to +/-Sqrt(2)/2).

The BFs for the full model do not tie. This may NOT be a bug because the interaction is parametrized differently under the anovaBF and generalTest models. However, two points on this. One, if #2 is not a bug than the full models should tie. Two, if I adjust the scale of the interaction using rscaleEffects to make the scale for the interaction equivalent across anovaBF and generalTestBF, the numbers still don't tie.

See attached for code.

BayesFactor Issue anovaBF versus generalTestBF.txt

[richarddmorey]

Thanks, I'll take a look.

[mattstern2]

Not a bug - so sorry Richard. Read on only if you're curious.

I coded the factors myself to test the generalTestBF model. I didn't realize, however, that this implies the regression approach is used. In this case, there is a single g parameters across all continuous variables (which you explicitly say is a bad prior for an ANOVA in your 2012 paper). Thus, it's not an equivalent model. I'm a bit embarrassed and will now close this issue. Sorry for the inconvenience.