Error in Convergence warning message (discrepancy between lme4)

[KennethEnevoldsen]

One of my student showed me the following discrepancy, which I have been able to reproduce:

It seems like the same model in lme4 and lmerTest both able to converge (while they are both singular fit), however lmerTest seem to throw an error message, which seems like a bug.

The lmerTest model throws the error: Model failed to converge with 1 negative eigenvalue: -1.6e+03

Even though the model obtains the convergence code: 0. Which according to the documentation means: convergence code, equal to zero for successful convergence

Here a shortened output using the latest version of lme4 and lmerTest from CRAN and run on MacOS Mojave Version 10.14.6:

model using lmerTest

> hrm2 <- lmerTest::lmer(changeHR_self ~
+                (HR_self + HR_other) * condition +
+                (condition|participant) +
+                (condition|group),
+              data = s4,
+              REML = F)
boundary (singular) fit: see ?isSingular
Warning message:
Model failed to converge with 1 negative eigenvalue: -1.6e+03 
> summary(hrm2)
Linear mixed model fit by maximum likelihood . t-tests use Satterthwaite's method ['lmerModLmerTest']
Formula: changeHR_self ~ (HR_self + HR_other) * condition + (condition |      participant) + 

(...)

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation matrix not shown by default, as p = 15 > 12.
Use print(x, correlation=TRUE)  or
    vcov(x)        if you need it

convergence code: 0
boundary (singular) fit: see ?isSingular

model using lme4

> hrm2_lme4 <- lme4::lmer(changeHR_self ~
+                          (HR_self + HR_other) * condition +
+                          (condition|participant) +
+                          (condition|group),
+                        data = s4,
+                        REML = F)
boundary (singular) fit: see ?isSingular
> summary(hrm2_lme4)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: changeHR_self ~ (HR_self + HR_other) * condition + (condition |      participant) + 

(...)

Correlation matrix not shown by default, as p = 15 > 12.
Use print(x, correlation=TRUE)  or
    vcov(x)        if you need it

convergence code: 0
boundary (singular) fit: see ?isSingular

[runehaubo]

This is probably because the test for negative eigenvalues is more accurate in lmerTest::lmer than in lme4::lmer. If I recall correctly the hessian is obtained by a rather crude method in lme4 (but it could have changed since I last looked) while the method used in lmerTest use numDeriv:: hessian which implements Richardson's extrapolation which is more accurate (see more pointers to this on page 17 in the lme4 manual that you link to under Theoretical Issues).

The Hessian computations in lmerTest are here so you can check the signs of the Hessian yourself in your example. The (crude) lme4-hessian is part of the model outputs so you can compare.

Cheers Rune

PS: Note that Satterthwaite and Kenward-Roger adjustments to t and F tests really only make sense with REML=TRUE.

[KennethEnevoldsen]

Thanks for the great answer it seems like the difference in the calculation of the hessian is indeed the differentiating factor. (Will also take a look further into this when I find the time)

It however does not seem to solve the discrepancy within lmerTest between the warning and the fact that the model did converge (according to the convergence code). I assume this should be addressed?

[runehaubo]

I think you misunderstand: The model did not converge in the sense that the optimizer stopped at a point which is strictly speaking not an optimum of the likelihood surface. This point may be close enough to the true MLE for your purposes but for lmerTest, which use derivatives of the likelihood function to compute Satterthwaite t and F tests, the model did not converge well enough.

Note that lmerTest use lme4::lmer to obtain the model fit so if lme4::lmer used the less crude (but computationally more expensive) Hessian evaluation which is used in lmerTest::lmer you would also be notified of the convergence failure by lme4::lmer. In both lme4 and lmerTest the trade-off in the effort used in determining convergence is chosen deliberately with lmerTest leaning toward higher accuracy at the expense of more computations.

[KennethEnevoldsen]

I see. However currently, beside the warning there is nothing saved in the model summary (at least what I can find) which indicate that the model did not converge. E.g. if the warning wasn't seen I would assume to model to have converged. I find that this is an issue? Preferably it would be ideal to know that lme4 considers that the model is converged while lmerTest does not.

[runehaubo]

No doubt that convergence checking and storage of this information in the model fit could be improved.

I don't think, however, that lme4 and lmerTest are in disagreement about whether the model converged or not; it didn't. It is just that lmer does not by default spend the computational time to determine that this is so. The optimizer used be lmer stops because it cannot improve on the model fit and so returns convergence code 0, but that is just the optimizer whose codes are not the ultimate truth (and generally rather lenient) . The post-fitting checking in lmer does not detect the lack of convergence - not because it "didn't not converge" but because by default lmer doesn't prioritize the computational recourses necessary to detect it. In this particular case lmerTest::lmer did prioritize enough recourses to detect the lack of convergence, but there is no guarantee that it will in every case.

If you want to spend the recourses yourself applying more convergence checks the lme4 manual describes how you can approach this under the entries convergence, checkConv, isSingular and rePCA. This would also lead you to determine the lack of convergence that lmerTest flags following the examples under these entries.

Your wish for better convergence information has been noted but beyond that I don't see a bug in the (existing) software so I'm going to close this issue here. Feel free to inquire about bugs another time.

Cheers Rune

[KennethEnevoldsen]

Thanks again Rune for the quick feedback. No there is no bug.

Best, Kenneth