Reporting model comparisons does not work due to effect size issue

[LukasWallrich]

Describe the bug Thanks for this wonderful package! I would have expected it to be able to report simple model comparison F-tests. However, they fail due to an issue with effect size calculation Error in .es_aov(as.data.frame(params), type, partial, generalized, ci, : No residuals data found - eta squared can only be computed for simple 'aov' models.

This could be resolved in various ways:

• if you don't want to support such reports at present, a clearer error message would be helpful
• if there is a way to get the reporting to work without an effect size, then it would be great to indicate how to do that in the error message

To Reproduce

mod1 <- lm(mpg ~ hp + wt, mtcars)
mod2 <- lm(mpg ~ hp * wt, mtcars)

report(anova(mod1, mod2))

Expected output

Sth like: The comparison suggests that the difference between the models is statistically significant, F(1, 28) = 14.1, p < .001

Specifiations (please complete the following information):

• Package Version: 0.3.5

[IndrajeetPatil]

Can reproduce:

library(report)

mod1 <- lm(mpg ~ hp + wt, mtcars)
mod2 <- lm(mpg ~ hp * wt, mtcars)

report(anova(mod1, mod2))
#> Error in .es_aov(as.data.frame(params), type, partial, generalized, ci, : No residuals data found - eta squared can only be computed for simple `aov` models.

^{Created on 2021-11-10 by the reprex package (v2.0.1)}

[mattansb]

Unfortunately in R anova objects are very data-sparse, so it is hard to figure out exactly what's going on just from that object.

Here are some alternatives:

mod1 <- lm(mpg ~ hp + wt, mtcars)
mod2 <- lm(mpg ~ hp * wt, mtcars)

(tp <- performance::test_performance(mod1, mod2))
#> Name | Model |     BF | Omega2 | p (Omega2) |    LR | p (LR)
#> ------------------------------------------------------------
#> mod1 |    lm |        |        |            |       |       
#> mod2 |    lm | 120.05 |   0.26 |      0.068 | 13.04 |  0.001
#> Models were detected as nested and are compared in sequential order.

report::report(tp)
#> We compared two models; lm (BF = 120.05; Omega2 = 0.26, p = 0.068; LR = 13.04, p = 0.001) and lm (BF = 120.05; Omega2 = 0.26, p = 0.068; LR = 13.04, p = 0.001).

@strengejacke @DominiqueMakowski Each model here incorrectly gets a BIC-BF - the same one. The "first" model should not get a BF.

effectsize::cohens_f_squared(mod1, model2 = mod2)
#> Cohen's f2 (partial) |           95% CI | R2_delta
#> --------------------------------------------------
#> 0.50                 | [0.13,      Inf] |     0.06
#> 
#> - One-sided CIs: upper bound fixed at (Inf).

^{Created on 2021-11-11 by the reprex package (v2.0.1)}

[LukasWallrich]

Thanks, @mattansb - that is helpful, yet supporting the more common anova tests would also be good. The heading often gives a good clue - I use the following as part of a function in a small helper package - not sure how often it would go wrong, though.

  # Likely from anova() of two lm() models
  if (stringr::str_detect(attributes(x)$heading[1], "Analysis of Variance Table")) {
    return(glue::glue("<em>F</em>({x$Df[2]}, {x$Res.Df[2]}) = {x$F[2] %>% round_(2)}, <em>p</em> {x$`Pr(>F)`[2] %>% fmt_p()}"))
  }

  # Likely from car::linearHypothesis()
  if (stringr::str_detect(attributes(x)$heading[1], "Linear hypothesis test")) {
    return(glue::glue("<em>F</em>({x$Df[2]}, {x$Res.Df[2]}) = {x$F[2] %>% round_(2)}, <em>p</em> {x$`Pr(>F)`[2] %>% fmt_p()}"))
  }

  # Likely from anova() of two lavaan models
  if (stringr::str_detect(attributes(x)$heading[1], "Chi-Squared Difference Test")) {
    return(glue::glue("<em>&chi;</em><sup>2</sup>({x$`Df diff`[2]}) = {x$`Chisq diff`[2] %>% round_(2)}, <em>p</em> {x$`Pr(>Chisq)`[2] %>% fmt_p()}"))
  }  

[mattansb]

@LukasWallrich Thanks for the suggestion - your code looks suitable for specific cases of anova() outputs,

However since there are no "official" requirements from the anova class, different developers write different information hidden in what-ever-formatted-text in the heading attribute (often nothing at all) and all the rest would be guess work depending on patterns of missing values in certain rows, number of rows, row names, column names, etc... This is precisely why we've added the in-house functions in performance that house more data and behave consistently and reliably.

[mattansb]

Closing this here.

@LukasWallrich If you have any idea how to reliably detect a wide-range of models compared using the anova() function, feel free to open an issue/pull-request on parameters.

[bwiernik]

Yes, in general, it is best to look at anova() as similar to print() and summary()--as an R output function intended for display of results in the console, not as an analytic or modeling fucntion