florianhartig / DHARMa

Diagnostics for HierArchical Regession Models
http://florianhartig.github.io/DHARMa/
201 stars 21 forks source link

Error with lme4 models with offset and re.form = NULL #335

Closed florianhartig closed 1 year ago

florianhartig commented 2 years ago

Reported by a user. Reproducible example:

library(DHARMa)
dat <- createData()

fit <- glmer(observedResponse ~ (1|group) + offset(Environment1), data = dat, family = "poisson")
summary(fit)
simulateResiduals(fit, plot = T, re.form = NULL)
florianhartig commented 2 years ago

OK, I assume that this is an lme4 problem, because if I check lme4 S3 functions for the model object

I don't see why simulate shouldn't work if predict works. I will probably have to send this to the lme4 crew. @bbolker - could you maybe have a look?

For the user - as a quick fix, you could use predict and then simulate by hand using the appropriate distribution.

bbolker commented 2 years ago

As the error message shows, this is a problem with trying to evaluate the offset term in an environment where it can't be found (the bane of my lme4-developing existence):

simulate(fit, re.form = NULL)

Error in offset(Environment1) : object 'Environment1' not found

PS it looks like offset is getting included in the random effects term when simulating, for some reason?? That might not be too hard to fix ...

florianhartig commented 2 years ago

Regarding the environment: OK, but I assume there is no fundamental reason for this to occur, so it's a programming bug on lme4 side, right? Sorry for adding to your pile / plight ;)

About the offset included in the REs - my understanding / expectation would have been that regardless of the re.form setting, the offset always needs to be included, because it's part of the model, so re.form only affects whether the REs should be included, but offset + fixed effects will always be included?

bbolker commented 2 years ago

I think https://github.com/lme4/lme4/commit/8c8ab5c95c9438f41667052602b101f7b78a52de should fix this ... although (a) I'm not sure it's 100% bulletproof (although I couldn't break it as easily as I thought I'd be able to) and (b) can't give a guarantee at the moment about when this will make it to CRAN (although I have my arm twisted; last CRAN submission was 7 May 2022, so should probably wait a couple of weeks at least)

florianhartig commented 2 years ago

Great, thanks! I'll leave this open for a moment to check with the next CRAN version!

florianhartig commented 1 year ago

This was fixed in glmmTMB 1.1.4 -> closed!