search

easystats / performance

:muscle: Models' quality and performance metrics (R2, ICC, LOO, AIC, BF, ...)
https://easystats.github.io/performance/
GNU General Public License v3.0
1.02k stars 90 forks source link

Possible bug: error is thrown saying check_model is not implemented, if model formula contains a ratio #591

Closed qdread closed 3 months ago

qdread commented 1 year ago

Here is a reprex of a potential bug. Ignore the fact that this is probably not a good model!

In this example, I fit two models that return identical output. In one model, I create a ratio of two variables within the model formula. In the other, I create a new column in the data frame by dividing the two columns, and then use that new name in the model formula. Oddly, check_model() throws a strange error in the first case: Error:check_model()not implemented for models of classlmerModyet. This doesn't seem like the desired behavior. Thanks in advance for looking into this!

reprex

library(lme4)
library(easystats)

model1 <- lmer(incidence/size ~ period + (1|herd), data = cbpp)

cbpp$proportion <- cbpp$incidence / cbpp$size

model2 <- lmer(proportion ~ period + (1|herd), data = cbpp)

all.equal(coefficients(model1), coefficients(model2)) # TRUE: they are the same

check_model(model1) # Error: `check_model()` not implemented for models of class `lmerMod` yet.
check_model(model2) # Works

session info

I just updated all easystats packages before running this, and verified the bug is still present using the latest CRAN versions of easystats packages.

R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] report_0.5.7       see_0.7.5          correlation_0.8.4  modelbased_0.8.6   effectsize_0.8.3   parameters_0.21.1  performance_0.10.3
 [8] bayestestR_0.13.1  datawizard_0.7.1   insight_0.19.2     easystats_0.5.2    lme4_1.1-31        Matrix_1.5-1      

loaded via a namespace (and not attached):
 [1] httr_1.4.4              splines_4.2.2           foreach_1.5.2           qqconf_1.3.1            assertthat_0.2.1       
 [6] memuse_4.2-3            robustbase_0.95-1       ggrepel_0.9.1           pillar_1.9.0            lattice_0.20-45        
[11] glue_1.6.2              minqa_1.2.5             colorspace_2.1-0        sandwich_3.0-2          pkgconfig_2.0.3        
[16] twosamples_2.0.0        xtable_1.8-4            mvtnorm_1.1-3           patchwork_1.1.2         scales_1.2.1           
[21] pracma_2.4.2            emmeans_1.8.2-090002    tibble_3.2.1            mgcv_1.8-41             generics_0.1.3         
[26] farver_2.1.1            ggplot2_3.4.2           TH.data_1.1-1           withr_2.5.0             opdisDownsampling_0.8.2
[31] cli_3.6.1               survival_3.4-0          magrittr_2.0.3          estimability_1.4.1      fansi_1.0.4            
[36] doParallel_1.0.17       nlme_3.1-160            MASS_7.3-58.1           benchmarkme_1.0.8       tools_4.2.2            
[41] lifecycle_1.0.3         multcomp_1.4-20         munsell_0.5.0           qqplotr_0.0.6           compiler_4.2.2         
[46] caTools_1.18.2          rlang_1.1.1             grid_4.2.2              nloptr_2.0.3            iterators_1.0.14       
[51] rstudioapi_0.14         bitops_1.0-7            labeling_0.4.2          boot_1.3-28             gtable_0.3.3           
[56] codetools_0.2-18        DBI_1.1.3               benchmarkmeData_1.0.4   R6_2.5.1                zoo_1.8-11             
[61] dplyr_1.0.10            utf8_1.2.3              parallel_4.2.2          Rcpp_1.0.10             vctrs_0.6.2            
[66] DEoptimR_1.0-12         tidyselect_1.2.0        coda_0.19-4
strengejacke commented 1 year ago

This issue seems to be resolved in one of our dev-versions, at least I cannot reproduce this issue:

library(lme4)
#> Loading required package: Matrix
library(performance)

model1 <- lmer(incidence / size ~ period + (1 | herd), data = cbpp)
check_model(model1)
#> Not enough model terms in the conditional part of the model to check for
#>   multicollinearity.

Created on 2023-06-02 with reprex v2.0.2

Can you run easystats::install_latest() and see if the error persists?

qdread commented 1 year ago

Unfortunately the error persists for me after running install_latest().

Looking for newer package versions...

Installing following packages:

Package     |   Latest | Installed
----------------------------------
datawizard  |  0.7.1.9 |     0.7.1
bayestestR  | 0.13.1.1 |    0.13.1
performance |   0.10.4 |    0.10.3
effectsize  | 0.8.3.11 |     0.8.3
modelbased  |  0.8.6.3 |     0.8.6
see         |  0.7.5.9 |     0.7.5
report      |  0.5.7.4 |     0.5.7

I ran the code exactly as in your post and got the same error that check_model() is not yet implemented for models of that class. Thanks for looking into this!

strengejacke commented 1 year ago

Can you try again and include your session info?

library(lme4)
#> Loading required package: Matrix
library(performance)

model1 <- lmer(incidence / size ~ period + (1 | herd), data = cbpp)
check_model(model1)
#> Not enough model terms in the conditional part of the model to check for
#>   multicollinearity.


sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.0 (2023-04-21 ucrt)
#>  os       Windows 10 x64 (build 19045)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-06-02
#>  pandoc   3.1.1 @ C:/Users/mail/AppData/Local/Pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  bayestestR          0.13.1.1 2023-06-01 [1] https://easystats.r-universe.dev (R 4.3.0)
#>  benchmarkme         1.0.8    2022-06-12 [1] CRAN (R 4.3.0)
#>  benchmarkmeData     1.0.4    2020-04-23 [1] CRAN (R 4.3.0)
#>  bitops              1.0-7    2021-04-24 [1] CRAN (R 4.3.0)
#>  boot                1.3-28.1 2022-11-22 [2] CRAN (R 4.3.0)
#>  caTools             1.18.2   2021-03-28 [1] CRAN (R 4.3.0)
#>  cli                 3.6.1    2023-03-23 [1] CRAN (R 4.3.0)
#>  codetools           0.2-19   2023-02-01 [2] CRAN (R 4.3.0)
#>  colorspace          2.1-0    2023-01-23 [1] CRAN (R 4.3.0)
#>  datawizard          0.7.1.9  2023-05-30 [1] https://easystats.r-universe.dev (R 4.3.0)
#>  DEoptimR            1.0-13   2023-05-02 [1] CRAN (R 4.3.0)
#>  digest              0.6.31   2022-12-11 [1] CRAN (R 4.3.0)
#>  doParallel          1.0.17   2022-02-07 [1] CRAN (R 4.3.0)
#>  dplyr               1.1.2    2023-04-20 [1] CRAN (R 4.3.0)
#>  evaluate            0.21     2023-05-05 [1] CRAN (R 4.3.0)
#>  fansi               1.0.4    2023-01-22 [1] CRAN (R 4.3.0)
#>  farver              2.1.1    2022-07-06 [1] CRAN (R 4.3.0)
#>  fastmap             1.1.1    2023-02-24 [1] CRAN (R 4.3.0)
#>  foreach             1.5.2    2022-02-02 [1] CRAN (R 4.3.0)
#>  fs                  1.6.2    2023-04-25 [1] CRAN (R 4.3.0)
#>  generics            0.1.3    2022-07-05 [1] CRAN (R 4.3.0)
#>  ggplot2             3.4.2    2023-04-03 [1] CRAN (R 4.3.0)
#>  ggrepel             0.9.3    2023-02-03 [1] CRAN (R 4.3.0)
#>  glue                1.6.2    2022-02-24 [1] CRAN (R 4.3.0)
#>  gtable              0.3.3    2023-03-21 [1] CRAN (R 4.3.0)
#>  htmltools           0.5.5    2023-03-23 [1] CRAN (R 4.3.0)
#>  httr                1.4.6    2023-05-08 [1] CRAN (R 4.3.0)
#>  insight             0.19.2   2023-05-30 [1] https://easystats.r-universe.dev (R 4.3.0)
#>  iterators           1.0.14   2022-02-05 [1] CRAN (R 4.3.0)
#>  knitr               1.43     2023-05-25 [1] CRAN (R 4.3.0)
#>  labeling            0.4.2    2020-10-20 [1] CRAN (R 4.3.0)
#>  lattice             0.21-8   2023-04-05 [1] CRAN (R 4.3.0)
#>  lifecycle           1.0.3    2022-10-07 [1] CRAN (R 4.3.0)
#>  lme4              * 1.1-33   2023-04-25 [1] CRAN (R 4.3.0)
#>  magrittr            2.0.3    2022-03-30 [1] CRAN (R 4.3.0)
#>  MASS                7.3-60   2023-05-04 [1] CRAN (R 4.3.0)
#>  Matrix            * 1.5-4.1  2023-05-18 [1] CRAN (R 4.3.0)
#>  memuse              4.2-3    2023-01-24 [1] CRAN (R 4.3.0)
#>  mgcv                1.8-42   2023-03-02 [1] CRAN (R 4.3.0)
#>  minqa               1.2.5    2022-10-19 [1] CRAN (R 4.3.0)
#>  munsell             0.5.0    2018-06-12 [1] CRAN (R 4.3.0)
#>  nlme                3.1-162  2023-01-31 [2] CRAN (R 4.3.0)
#>  nloptr              2.0.3    2022-05-26 [1] CRAN (R 4.3.0)
#>  opdisDownsampling   0.8.2    2022-05-24 [1] CRAN (R 4.3.0)
#>  patchwork           1.1.2    2022-08-19 [1] CRAN (R 4.3.0)
#>  performance       * 0.10.4   2023-06-02 [1] local
#>  pillar              1.9.0    2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig           2.0.3    2019-09-22 [1] CRAN (R 4.3.0)
#>  pracma              2.4.2    2022-09-22 [1] CRAN (R 4.3.0)
#>  purrr               1.0.1    2023-01-10 [1] CRAN (R 4.3.0)
#>  qqconf              1.3.2    2023-04-14 [1] CRAN (R 4.3.0)
#>  qqplotr             0.0.6    2023-01-25 [1] CRAN (R 4.3.0)
#>  R.cache             0.16.0   2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3         1.8.2    2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo                1.25.0   2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils             2.12.2   2022-11-11 [1] CRAN (R 4.3.0)
#>  R6                  2.5.1    2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp                1.0.10   2023-01-22 [1] CRAN (R 4.3.0)
#>  reprex              2.0.2    2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang               1.1.1    2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown           2.21     2023-03-26 [1] CRAN (R 4.3.0)
#>  robustbase          0.95-1   2023-03-29 [1] CRAN (R 4.3.0)
#>  scales              1.2.1    2022-08-20 [1] CRAN (R 4.3.0)
#>  see                 0.7.5.9  2023-06-02 [1] local
#>  sessioninfo         1.2.2    2021-12-06 [1] CRAN (R 4.3.0)
#>  styler              1.10.0   2023-05-24 [1] CRAN (R 4.3.0)
#>  tibble              3.2.1    2023-03-20 [1] CRAN (R 4.3.0)
#>  tidyselect          1.2.0    2022-10-10 [1] CRAN (R 4.3.0)
#>  twosamples          2.0.0    2022-07-12 [1] CRAN (R 4.3.0)
#>  utf8                1.2.3    2023-01-31 [1] CRAN (R 4.3.0)
#>  vctrs               0.6.2    2023-04-19 [1] CRAN (R 4.3.0)
#>  withr               2.5.0    2022-03-03 [1] CRAN (R 4.3.0)
#>  xfun                0.39     2023-04-20 [1] CRAN (R 4.3.0)
#>  yaml                2.3.7    2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] C:/Users/mail/AppData/Local/R/win-library/4.3
#>  [2] C:/Program Files/R/R-4.3.0/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Created on 2023-06-02 with reprex v2.0.2

qdread commented 1 year ago 
> sessioninfo::session_info()
─ Session info ──────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.2.2 (2022-10-31 ucrt)
 os       Windows 10 x64 (build 19044)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_United States.utf8
 ctype    English_United States.utf8
 tz       America/New_York
 date     2023-06-02
 rstudio  2023.03.0+386 Cherry Blossom (desktop)
 pandoc   2.18 @ C:\\Users\\qdread\\AppData\\Local\\Pandoc\\pandoc.exe

─ Packages ──────────────────────────────────────────────────────────────────────────────────
 package     * version  date (UTC) lib source
 bayestestR    0.13.1.1 2023-06-01 [1] https://easystats.r-universe.dev (R 4.2.3)
 boot          1.3-28   2021-05-03 [2] CRAN (R 4.2.2)
 cli           3.6.1    2023-03-23 [1] CRAN (R 4.2.3)
 datawizard    0.7.1.9  2023-05-30 [1] https://easystats.r-universe.dev (R 4.2.3)
 insight       0.19.2   2023-05-23 [1] CRAN (R 4.2.3)
 lattice       0.20-45  2021-09-22 [2] CRAN (R 4.2.2)
 lme4        * 1.1-31   2022-11-01 [1] CRAN (R 4.2.2)
 MASS          7.3-58.1 2022-08-03 [2] CRAN (R 4.2.2)
 Matrix      * 1.5-1    2022-09-13 [2] CRAN (R 4.2.2)
 minqa         1.2.5    2022-10-19 [1] CRAN (R 4.2.2)
 nlme          3.1-160  2022-10-10 [2] CRAN (R 4.2.2)
 nloptr        2.0.3    2022-05-26 [1] CRAN (R 4.2.2)
 performance * 0.10.4   2023-06-02 [1] https://easystats.r-universe.dev (R 4.2.3)
 Rcpp          1.0.10   2023-01-22 [1] CRAN (R 4.2.3)
 rstudioapi    0.14     2022-08-22 [1] CRAN (R 4.2.2)
 sessioninfo   1.2.2    2021-12-06 [1] CRAN (R 4.2.2)

 [1] C:/Users/qdread/AppData/Local/R/win-library/4.2
 [2] C:/Program Files/R/R-4.2.2/library
elichten commented 1 year ago

My student and I are having similar issues with check_model(). I have performance v. 0.10.2, and my student has v. 0.10.4. We are both using Macs and RStudio (2023.03.1+446 for me, 2023.06.0+421 for my student). My student updated R yesterday (and easystats today), but I am running R v. 4.2.2.

bbolker commented 7 months ago

For future reference, "Error: check_model() not implemented for models of class XX yet" is the generic message when anything goes wrong with extracting information from the model (e.g. https://github.com/easystats/performance/issues/678). With development versions of packages, options(easystats_errors = TRUE) should return the original message and make debugging easier (see https://github.com/easystats/performance/issues/691) ...

strengejacke commented 7 months ago

Furthermore, since June 2023, we also added more informative error messages in various places. This still might not capture every possible error, but as Ben wrote, setting options(easystats_errors = TRUE) should return the original error message in all (for users) relevant occasions.

strengejacke commented 3 months ago

Fixed in #751

library(lme4)
#> Loading required package: Matrix
library(easystats)
#> # Attaching packages: easystats 0.7.2.3
#> ✔ bayestestR  0.13.2.5    ✔ correlation 0.8.5    
#> ✔ datawizard  0.12.0      ✔ effectsize  0.8.9    
#> ✔ insight     0.20.2.1    ✔ modelbased  0.8.8    
#> ✔ performance 0.12.0.11   ✔ parameters  0.22.0.2 
#> ✔ report      0.5.9       ✔ see         0.8.4.7
cbpp$proportion <- cbpp$incidence / cbpp$size

model1 <- lmer(incidence/size ~ period + (1|herd), data = cbpp)
model2 <- lmer(proportion ~ period + (1|herd), data = cbpp)

all.equal(coefficients(model1), coefficients(model2)) # TRUE: they are the same
#> [1] TRUE

check_model(model1)

check_model(model2)

Created on 2024-07-14 with reprex v2.1.1