search

eric-pedersen / mixed-effect-gams

Repo for tutorial/paper on mixed-effect GAMs
MIT License
122 stars 32 forks source link

Switched to using differences in deviances to compare models #41

Closed eric-pedersen closed 5 years ago

eric-pedersen commented 5 years ago

To deal with issue #38 I've attempted to change to using deviances for out-of-sample fits rather than RMSE in the zooplankton example. I'm still a bit uncertain if I'm using the right form of deviance to compare model fits though. Simon talks about both scaled and unscaled deviances in the book; I've focused on unscaled deviance, but is that appropriate here? @dill, I think we need your input on this. I'm getting a bit out of my depth on the statistical theory of glms.

dill commented 5 years ago

It seems like from Wood (2017) that the scaling is only really used for getting to the Chi-sq test statistic, so I think we are fine to not scale.

Looks to be from the out-of-sample deviance that the difference between models 4 and 5 (which I guess are renamed now) are marginal in most cases, and one the "intercept" model does better (what is happening with K. cochlearis??), but I guess our point is that in the cases where it does matter we have that extra flexibility.

dill commented 5 years ago

I also went through the code that generates this and it looks good to me, btw.