Open rsbivand opened 3 years ago
Does this happen with other backends? If not, it may be specific to the scipy.sparse.SuperLU solver.
Could you post the gal? I'm happy to generate fake data to correspond to it.
Does
spinv()
go dense on return?
It should remain sparse, since the input remains sparse at line 192, but I can instrument this.
Does
spinv()
go dense on return?It should remain sparse, since the input remains sparse at line 192, but I can instrument this.
If the input is a sparse object, spinv()
will return a sparse object, as can be seen on sputils.py
, lines 233-237.
if SP.issparse(a):
ai = SPla.inv(a)
else:
ai = la.inv(a)
return ai
However, depending on the matrix, its inverse will be dense, albeit still a sparse object.
Yes, (I - \lambda W)^{-1}$
is dense by definition, so spinv()
returns a sparse matrix with all elements non-zero. @ljwolf - here is the compressed GAL file.
large_nb.zip
I think that for method='LU'
, the variance covariance matrix for the betas is all that can be returned if N is large. An LR test of OLS against ML tests for the inclusion of \lambda.
Does
spinv()
go dense on return?It should remain sparse, since the input remains sparse at line 192, but I can instrument this.
If the input is a sparse object,
spinv()
will return a sparse object, as can be seen onsputils.py
, lines 233-237.if SP.issparse(a): ai = SPla.inv(a) else: ai = la.inv(a) return ai
However, depending on the matrix, its inverse will be dense, albeit still a sparse object.
Ahh, I see.... If that's the case, then, @rsbivand's suggestion is very reasonable, I think. We should avoid the rest of the baseclass (e.g, avoid taking the spinv(ai)
) for folks who request method="LU"
.
My hope for implementing LU
was that it should avoid instantiating a dense inverse, both during the optimization step and elsewhere. The ord
and full
methods immediately instantiate w.full()
.
We'd need to do this as well for ml_lag.py
, which is also densified in all methods except LU
.
Yes,
(I - \lambda W)^{-1}$
is dense by definition, sospinv()
returns a sparse matrix with all elements non-zero. @ljwolf - here is the compressed GAL file. large_nb.zipI think that for
method='LU'
, the variance covariance matrix for the betas is all that can be returned if N is large. An LR test of OLS against ML tests for the inclusion of \lambda.
Thank you for the file! Will use for the test case for this.
The way around this for large N may be an FD Hessian if one (that works) is available in a Python package you already use. Maybe from a numerical optimizer? Unlike the line search, it starts from the \lambda and \beta we have, and searches close to them to estimate the joint distribution shape. Here we only need \lamba (and maybe \sigma^2) because there are no interactions with the var-covar of the \betas.
While you are looking at that bit of ML_Error, maybe consider adding the Pace-Lesage Hausman test at least for 'full'
and 'ord'
. For 'LU'
, it is possible to approximate by powering \lambda W x (if powers guarantee a dampened series, no explosions please!) for an x vector or matrix, as the powered W never gets stored, only cumulated in the target vector (up to an arbitrary cutoff). Unfortunately, as it tests for differences between OLS and *_error \betas, it often finds mis-specification.
I was trying to fit an ML error model with 71' observations, but memory use grew very quickly. This may be because some kind of parallelization comes into play (it shouldn't), but it feels more like the weights matrix going dense. The messages before I killed the process were:
I think the sparse weights matrix is CSR not CSC. Is the problem in the densifying of the variance covariance matrix? About line https://github.com/pysal/spreg/blob/c6d97c1b31bb4997a7c65b3f6b35a1ce3e11282a/spreg/ml_error.py#L244 ? Could a finite difference Hessian help? Does
spinv()
go dense on return?