Closed vhmedina closed 8 months ago
Hi @vhmedina ,
Thanks for your question / issue. There are two reasons you see this additional shrinkage in the random effects when using deepregression
:
mgcv
, the optimization will make a difference and in this case induce additional shrinkage. fac_processor
defines a ridge-penalized linear (categorical) effect. While this will induce the same "penalty" as a normal distribution assumption for random effects, the amount of penalization is fixed. We treat it as a hyperparameter (la
) and you could, e.g., grid-search over different values for this hyperparameter (in contrast, mgcv
estimates this penalty parameter implicitly by estimating the random effect variances).Regarding point 2: We will hopefully soon have an update that allows to automatically learn smoothing parameters and random effects variances.
Let me know if you have any further questions.
HTH, David
Many thanks for the reply, @davidruegamer. This is really helpful.
I played around a bit with grid-search over different values for lambda. The closest combination I got was when la=0
(no regularization). It improved slightly when changing the optimizer to SGD with larger momentum, but I couldn't reach the mgcv
results. Thanks again!
Interesting. Thanks for sharing your results! I think it might not (always) be possible to reach the mgcv
result due to the implicit regularization. Switching from Adam (i.e., parameter-specific learning rates) to SGD might in fact mitigate the problem as the implicit shrinkage is less specific, but chances are still small that one ends up with the same amount of penalization.
Hi!
When comparing the estimations of random intercepts in a simple simulation between
mgcv
anddeepregression
, I obtain different results. The results withdeepregression
tend to be more shrank than inmgcv
.Am I missing something? Please find a reproducible example below. Many thanks!
sessionInfo()
is the following