easystats / performance

:muscle: Models' quality and performance metrics (R2, ICC, LOO, AIC, BF, ...)
https://easystats.github.io/performance/
GNU General Public License v3.0
1.02k stars 89 forks source link

Transformed responses in performance_aic #773

Open JoakimStefansson opened 1 day ago

JoakimStefansson commented 1 day ago

Background

There are some gaps in the ability of performance_aic() to detect and correct transformed responses. It is of course very hard to cover every possible case, but the documentation does promise a lot:

performance_aic() correctly detects transformed response and, unlike stats::AIC(), returns the "corrected" AIC value on the original scale.

Suggested action

  1. I suggest the function should return a warning when we know something is up with the response variable, but the transformation can't be fully identified and corrected. I'm talking about the cases where insight::find_transformation returns NULL. something like: >The transformation and inverse-transformation functions could not be determined. The AIC has been determined without correction.

  2. If Insight::find_transformation==NULL should have this role of triggering warnings in other functions, it would be nice if it covered all suspicious cases. The regex any(grepl("I\\((.*)\\)", x)) is beyond me, but it seems a divided response is neither caught nor corrected. See below. Wouldn't it be ideal if the occurrence of anything that is not a number or a named object (or a transformation that is handled) would return a NULL transformation?

    
    a<-aov(formula=mpg/0.7 ~ hp, data=mtcars) # transformation is overt, and should be corrected
    b<-aov(formula=mpg_trans ~ hp, data=mtcars) # transformation is hidden in the data, and can't be corrected
    c<-aov(formula=I(mpg/0.7) ~ hp, data=mtcars) # same as a, but is identified as NULL transformation

find_transformation(a) # "identity" find_transformation(b) # "identity" find_transformation(c) # NULL

performance_aicc(a) # 204.923 performance_aicc(b) # 204.923 performance_aicc(c) # 204.923

strengejacke commented 1 day ago

Maybe similar to https://github.com/easystats/insight/issues/941. I think we can detect this in find_transformation(), but we can't have get_transformation() working here.