Open lezcano opened 3 weeks ago
I'd like to elaborate on more fundamental reasons why CG and GMRES are needed and where they may be useful.
torch.linalg.solve
: it directly computes the matrix inversion, which only works for dense and invertible matrices. It cannot solve general linear systems. It's also impractical when dealing with large-scale linear systems due to memory $O(n^2)$ and time complexity $O(n^3)$.In summary,
torch.linalg.solve
cannot solve. The specific implementation can always be optimized later; however, it is probably better to consider whether it's good to incorporate these new algorithms from a more strategic, long-term perspective for the development of PyTorch.
For generic systems we have lstsq
. For symmetric non-definite systems we have https://pytorch.org/docs/stable/generated/torch.linalg.ldl_factor.html.
That being said, I do agree that these solvers could potentially be of interest, but before adding them to PyTorch core, we would need to:
LinearOperator
.
I would expect this Python implementation to be quite a bit slower than
linalg.solve
. If this is the case, the applications of this would be to use it with custom operators, which PyTorch doesn't currently support natively (see https://github.com/pytorch/pytorch/issues/28341)