ENH linalg tools weighted orthogonalization

[josef-pkt]

(just a quick idea)

We orthogonalize in several cases one set from another set of variables. If we have weights, then we need to orthogonalize with respect to the measure defined by the weights.

I think in the WLS case we can just use the weighted variables, e.g. wexog.

usecase: hessian/EIM in GLM when "absorbing" some exog like categorical fixed effects. #5266 Once we have the hessian_factor, the hessian is similar to the one in WLS. We should be able to make it block-diagonal by reparameterization and compute the inverse by partitioned inverse or inverting just the block of interest.

(I don't have time right now to think about how and whether that actually works.)

The setup is similar to score_test for added variables in GLM when we use the partitioned inverse for the distribution of the score.

[josef-pkt]

just an analogue: orthogonal polynomials are orthogonal with respect to some distribution, but not orthogonal without weighting.

[josef-pkt]

It sounds kind of obvious, only I never thought about it in the more general context of GLM outside of WLS/IRLS version, i.e. this should apply locally to wherever we have hessian_factor structure of the hessian with respect to the exog params. (Other params, like dispersion in NB and GP, might need separate treatment, and orthogonalizing exog subsets won't be enough if we cannot separate out or orthogonalize the extra params as in two-part models, e.g. ZIGP?, I guess.) http://scs.math.yorku.ca/index.php/Statistics:_Frisch-Waugh-Lovell_and_GLS

[josef-pkt]

For the GLM absorbtion case #5266, we should get everything then as a result of the absorbing WLS/IRLS model. However, similar to how I extended score_tests and similar, we should just be able to use score_factor and hessian_factor to compute this directly from the nonlinear MLE.

Another question: How can we do this for GMM? Locally we have the same orthogonalization or partial inverse in GMM as in MLE, which I already use in the analogy between score_test and conditional moment tests.

Because Wald and LM inference is only local, we can use whatever approach for estimating the params is computationally efficient, and then use some similar approach to construct the partial inverse or orthogonalized hessian and cov_params for the non-absorbed variables.

[josef-pkt]

(I don't remember how these linalg tools fit together)

Another helper function drop columns from second matrix that are collinear with first matrix (i.e. perfectly predicted, zero (+ floating point noise) residual var/sum of squares. This is more limited than full orthogonalizing. related #5364 and maybe other issues (This is similar to a partitioned matrix (2 submatrices) version of sequentially dropping collinear columns as with QR)

current use case: widely linear estimation for complex improper signals, requires augmenting exog with exog.conj() #3528 What if some exog.conj() are collinear with exog, e.g. if some exog are real (have zero imaginary part)? (I didn't find any references specifically for this.)

cases: X2 | X1, given X1 we want a transformed X2 with given properties:

• no perfect collinearity combined [X1, X2] has full rank -> orthogonalize T(X2) orthogonal to X1 (residual of projection)
  • maybe variant make T(X2) itself an orthogonal matrix
• collinear and orthogonalize -> complement of X2 given X1, only keep some orthogonal basis vectors spanning complement of X2 | X1 (result will have nonsingular combined [X1, T(X2)] if X1 is nonsingular.)
• collinear and don't orthogonalize -> drop columns of X2 that are collinear with X1 (case in this comment). This does not remove collinearity/singularity within X2 (and cases where an X2 column is collinear with a combination of X1, X2 column). Maybe better add a matrix rank check for the final combined result.

[josef-pkt]

another tool: sequential orthogonalization I guess that's just QR or cholesky, but right now I don't remember the details

https://github.com/statsmodels/statsmodels/pull/6268#issuecomment-2128031355