search

easystats / insight

:crystal_ball: Easy access to model information for various model objects
https://easystats.github.io/insight/
GNU General Public License v3.0
394 stars 39 forks source link

CRAN found a revdep problem with the `insight` package (details below) #679

Closed strengejacke closed 1 year ago

strengejacke commented 1 year ago

CRAN found a revdep problem with the insight package (details below). @strengejacke, I think you are the maintainer. I need to be in an online meeting for most of today, but will try get to this in the evening if someone else hasn't already. 

Package: insight
Check: tests
New result: ERROR
Running ‘testthat.R’ [82s/82s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
if (require("testthat")) {
   +   library(insight)
   + 
   +   is_dev_version <- length(strsplit(packageDescription("insight")$Version, "\\.")[[1]]) > 3
   + 
   +   if (is_dev_version) {
   +     Sys.setenv("RunAllinsightTests" = "yes")
   +   } else {
   +     Sys.setenv("RunAllinsightTests" = "no")
   +   }
   +   si <- [Sys.info](http://sys.info/)()
   + 
   +   osx <- tryCatch(
   +     {
   +       if (!is.null(si["sysname"])) {
   +         si["sysname"] == "Darwin" || grepl("^darwin", R.version$os)
   +       } else {
   +         FALSE
   +       }
   +     },
   +     error = function(e) {
   +       FALSE
   +     }
   +   )
   + 
   +   solaris <- tryCatch(
   +     {
   +       if (!is.null(si["sysname"])) {
   +         grepl("SunOS", si["sysname"], ignore.case = TRUE)
   +       } else {
   +         FALSE
   +       }
   +     },
   +     error = function(e) {
   +       FALSE
   +     }
   +   )
   + 
   +   # disable / enable if needed
   +   if (.Platform$OS.type == "unix" && is_dev_version) {
   +     Sys.setenv("RunAllinsightStanTests" = "yes")
   +   } else {
   +     Sys.setenv("RunAllinsightStanTests" = "no")
   +   }
   + 
   +   # if (!osx && !solaris) {
   +   #   test_check("insight")
   +   # }
   + 
   +   test_check("insight")
   + }
   Loading required package: testthat
   boundary (singular) fit: see help('isSingular')
   Some of the variables were in matrix-format - probably you used
     `scale()` on your data?
     If so, and you get an error, please try `datawizard::standardize()` to
     standardize your data.
   Confidence intervals are not yet supported for models of class `mlm`.

   Iteration 1 - deviance = 39.74973 - criterion = 0.8590917
   Iteration 2 - deviance = 10.50328 - criterion = 2.758244
   Iteration 3 - deviance = 9.231325 - criterion = 0.1363107
   Iteration 4 - deviance = 9.227742 - criterion = 0.0003840654
   Iteration 5 - deviance = 9.227742 - criterion = 3.446463e-09
   converged
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.

   Re-fitting to get Hessian

   Re-fitting to get Hessian

   boundary (singular) fit: see help('isSingular')
   boundary (singular) fit: see help('isSingular')
   [ FAIL 4 | WARN 51 | SKIP 16 | PASS 2516 ]

   ══ Skipped tests ═══════════════════════════════════════════════════════════════
   • On CRAN (6)
   • isFALSE(run_stan) is TRUE (5)
   • works interactively (5)

   ══ Failed tests ════════════════════════════════════════════════════════════════
   ── Failure ('test-glmmTMB.R:927'): get_predicted ───────────────────────────────
   `x` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...
   ── Failure ('test-glmmTMB.R:928'): get_predicted ───────────────────────────────
   `y` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...
   ── Failure ('test-glmmTMB.R:936'): get_predicted ───────────────────────────────
   `x` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...
   ── Failure ('test-glmmTMB.R:937'): get_predicted ───────────────────────────────
   `y` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...

   [ FAIL 4 | WARN 51 | SKIP 16 | PASS 2516 ]
   Error: Test failures
   Execution haltedPackage: insight
Check: tests
New result: ERROR
Running ‘testthat.R’ [82s/82s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
if (require("testthat")) {
   +   library(insight)
   + 
   +   is_dev_version <- length(strsplit(packageDescription("insight")$Version, "\\.")[[1]]) > 3
   + 
   +   if (is_dev_version) {
   +     Sys.setenv("RunAllinsightTests" = "yes")
   +   } else {
   +     Sys.setenv("RunAllinsightTests" = "no")
   +   }
   +   si <- [Sys.info](http://sys.info/)()
   + 
   +   osx <- tryCatch(
   +     {
   +       if (!is.null(si["sysname"])) {
   +         si["sysname"] == "Darwin" || grepl("^darwin", R.version$os)
   +       } else {
   +         FALSE
   +       }
   +     },
   +     error = function(e) {
   +       FALSE
   +     }
   +   )
   + 
   +   solaris <- tryCatch(
   +     {
   +       if (!is.null(si["sysname"])) {
   +         grepl("SunOS", si["sysname"], ignore.case = TRUE)
   +       } else {
   +         FALSE
   +       }
   +     },
   +     error = function(e) {
   +       FALSE
   +     }
   +   )
   + 
   +   # disable / enable if needed
   +   if (.Platform$OS.type == "unix" && is_dev_version) {
   +     Sys.setenv("RunAllinsightStanTests" = "yes")
   +   } else {
   +     Sys.setenv("RunAllinsightStanTests" = "no")
   +   }
   + 
   +   # if (!osx && !solaris) {
   +   #   test_check("insight")
   +   # }
   + 
   +   test_check("insight")
   + }
   Loading required package: testthat
   boundary (singular) fit: see help('isSingular')
   Some of the variables were in matrix-format - probably you used
     `scale()` on your data?
     If so, and you get an error, please try `datawizard::standardize()` to
     standardize your data.
   Confidence intervals are not yet supported for models of class `mlm`.

   Iteration 1 - deviance = 39.74973 - criterion = 0.8590917
   Iteration 2 - deviance = 10.50328 - criterion = 2.758244
   Iteration 3 - deviance = 9.231325 - criterion = 0.1363107
   Iteration 4 - deviance = 9.227742 - criterion = 0.0003840654
   Iteration 5 - deviance = 9.227742 - criterion = 3.446463e-09
   converged
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.
   Confidence intervals are not yet supported for models of class
     `multinom`.

   Re-fitting to get Hessian

   Re-fitting to get Hessian

   boundary (singular) fit: see help('isSingular')
   boundary (singular) fit: see help('isSingular')
   [ FAIL 4 | WARN 51 | SKIP 16 | PASS 2516 ]

   ══ Skipped tests ═══════════════════════════════════════════════════════════════
   • On CRAN (6)
   • isFALSE(run_stan) is TRUE (5)
   • works interactively (5)

   ══ Failed tests ════════════════════════════════════════════════════════════════
   ── Failure ('test-glmmTMB.R:927'): get_predicted ───────────────────────────────
   `x` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...
   ── Failure ('test-glmmTMB.R:928'): get_predicted ───────────────────────────────
   `y` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...
   ── Failure ('test-glmmTMB.R:936'): get_predicted ───────────────────────────────
   `x` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...
   ── Failure ('test-glmmTMB.R:937'): get_predicted ───────────────────────────────
   `y` (`actual`) not equal to `z` (`expected`).

        actual       | expected                     
    [1] 0.2400130397 - 0.3747267975 [1]             
    [2] 0.9726394192 - 1.1037499976 [2]             
    [3] 0.2400130397 - 0.3747267975 [3]             
    [4] 0.4613929713 - 0.5692223933 [4]             
    [5] 0.2400130397 - 0.3747267975 [5]             
    [6] 0.6362097213 - 0.6854377778 [6]             
    [7] 0.3344547486 - 0.4168940552 [7]             
    [8] 0.0059284577 - 0.0178449327 [8]             
    [9] 0.0902188470 - 0.1367765053 [9]             
   [10] 0.9726394192 - 1.1037499976 [10]            
    ... ...            ...          and 240 more ...

   [ FAIL 4 | WARN 51 | SKIP 16 | PASS 2516 ]
   Error: Test failures
   Execution halted

Originally posted by @mebrooks in https://github.com/glmmTMB/glmmTMB/issues/873#issuecomment-1308426818

DominiqueMakowski commented 1 year ago

Seems related to https://github.com/easystats/modelbased/issues/217

strengejacke commented 1 year ago

Yes, they changed something in the truncated_poisson family. I found out following (copying my response from the related glmmTMB issue):

I think the problem is that for zero-inflation models, we assume that the predicted response is mu * (1 - zi_prob). We also compute predictions on the link-scale, then backtransform. However, this behaviour seems to have changed:

library(glmmTMB)
library(insight)
data("fish")
  m1 <- glmmTMB(
  count ~ child + camper + (1 | persons),
  ziformula = ~ child + camper + (1 | persons),
  data = fish,
  family = truncated_poisson()
)

predict(m1, type = "response") |> head()
#> [1] 0.3747268 1.1037500 0.3747268 0.5692224 0.3747268 0.6854378

(link_inverse(m1)(predict(m1, type = "link")) * (1 - as.vector(predict(m1, type = "zprob")))) |> head()
#> [1] 0.2400130 0.9726394 0.2400130 0.4613930 0.2400130 0.6362097

linv <- family(m1)$linkinv
(linv(predict(m1, type = "link")) * (1 - as.vector(predict(m1, type = "zprob")))) |> head()
#> [1] 0.2400130 0.9726394 0.2400130 0.4613930 0.2400130 0.6362097

My question is, why is predicting the response no longer equal to predicting the count model on the link-scale, back-transform via link-inverse and multiply with the predicted zero-inflation-probability?

strengejacke commented 1 year ago

This issue also might affect the way we calculate CIs for truncated families (via the simulations)