Open Gabriel-Kissin opened 3 months ago
The core of the issue appears to be related to the algorithm used to find the inverse matrix. statsmodels uses by default the Moore-Penrose pseudoinverse to solve the least squares problem (link). When telling it to use the QR algorithm (method = "qr"
instead of method = "pinv"
) it fits correctly. See the legend in the plot above for comparison of the betas learnt by the different methods.
My suggestion then would be for default behaviour to intelligently select an algorithm based on aspects of the data which make one more suited than the other, so that it can be used out of the box like with the sklearn one.
It's very unlikely that we ever change this. QR does not have regularization in the case of numerically singular exog. (What rcond pinv threshold would be better is still open. there are problems if the condition number is around 1e15 (or 1e-15 if inverse is used), but it can go either way, regularization too soon or too late)
also still open: adding internally some transformation, but that was more in context of nonlinear models like exp in Poisson or Logit. #1131
your last OLS has the warning in the summary
Notes:
[1] ...
[2] The condition number is large, 2.55e+15. This might indicate that there are
strong multicollinearity or other numerical problems.
In general, the linear algebra with a condition number this high looses almost all numerical precision. It might work in some cases, but would produce very low precision in other cases. If the issue is a scaling issue, then the user needs to provide reasonably scaled variables. If the issue is inherent multicollinearity, then the user would be better of to treat is properly, e.g. choose a penalized estimator or drop variables. statsmodels is currently not doing anything for this except for the pinv in linear regression.
The problem in this case could partially be alleviated by treating the constant differently, but this would not help if the high condition number comes from high multicollinearity of the (slope) explanatory variables. That means that we would just be shifting the problems around to other cases. (another popular case that fails is power polynomials when the underlying x has large values.)
stats_ols_fitted.model.exog[:5]
array([[1.00000000e+00, 1.90000000e+14],
[1.00000000e+00, 1.96666667e+14],
[1.00000000e+00, 2.03333333e+14],
[1.00000000e+00, 2.10000000e+14],
[1.00000000e+00, 2.16666667e+14]])
Thank you @josef-pkt for your thorough response. I understand the issues that you describe with making code changes to fix this.
The only other thought I have then would be to add a note to the OLS documentation pointing out that
While the example I posted is contrived, I came across this issue using a real life dataset, and it took me a while to boil down the issue to what I posted above. Having a note about this issue in the documentation could potentially be very useful to others.
Thank you for all your hard work on statsmodels (and scipy.stats)!
We have relate issues
and more on tools to diagnose multicollinearity
quote of Welch cited in #9068 " Even with a vast arsenal of diagnostics, it is very hard to write down rules that can be used to guide a data analysis. So much is really subjective and subtle. "
It's work and it's not clear how much the software can take over from the user.
We try to provide more, both for numerical problems and cases where model does not look appropriate for the data, but it's slow going because there so many possibilities that can go "wrong".
For the specific scaling issue here this might help for a good warning message https://github.com/statsmodels/statsmodels/issues/2378#issuecomment-257031644
Describe the bug
When using large values for linear regression, statsmodels fails to correctly model the data. It seems to want the line to pass through the intercept when it shouldn't (despite
add_constant
being used).Here is the plot produced by the code below:![image](https://github.com/statsmodels/statsmodels/assets/118690308/7cd1da42-2626-42a2-95c7-2649c6537ffa)
For the first three lines, statsmodels works as expected. For larger values, the lines aren't passing through the corresponding points, instead wanting to go close to the intercept. This is because the coefficient being learned for
const
is always very small.Note that the black dotted lines produced by sklearn always fit the points as expected. The intercept and coefficient sklearn produces are the same as the closed-form solution, which statsmodels should be capable of reproducing. These numbers aren't large enough to cause an overflow on 32-bit precision.
Note: A simple workaround is to scale the data before fitting. Linear regression generally doesn't use GD, so scaling shouldn't be necessary though.
Code Sample
If the issue has not been resolved, please file it in the issue tracker.
Expected Output
Correctly fit model!
Output of
import statsmodels.api as sm; sm.show_versions()