[feature request] normest : matrix 2-norm estimation

[kose-y]

I think it would be great to have a method to estimate a matrix norm. This method is useful for iterative optimization algorithms, as region of parameters for theoretically guaranteed convergence is often defined in terms of 2-norm of the data matrix.

The following are relevant implementations.

numpy.linalg.norm: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html : frobenius, nuclear, l-infinity norm, l0, l1, l2, "-1"-norm, "-2"-norm (-1, -2-norms are not mathematical norms)

scipy.linalg.norm: https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html

MATLAB: https://mathworks.com/help/matlab/ref/normest.html : this one estimates matrix 2-norm.

cc @vishwakftw @SsnL @jianyuh

[y0ast]

Is there anything missing in the current torch.norm() ?

http://pytorch.org/docs/0.3.1/torch.html#torch.norm

[kose-y]

@y0ast torch.norm() simply computes rowwise vector norm. What I mean here is the matrix norm. See https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html for some examples. Some are straightforward to implement, but l2-norm, which is fairly important in my field (statistics) requires efficient computation of largest singular value by the power iteration.

[kose-y]

Power iteration will be much more useful for sparse matrices, as SVD is not feasible for sparse matrices.

[jonaskohler]

torch.norm() already gives you the frobenius norm of a matrix since it vectorizes the input and then computes the l2-norm. Other than that, I agree that it would be nice to have a scipy/numpy like implementation of induced operator norms.

[dragen1860]

@JoelMarcey . I feel confused about how to use torch.norm to compute vector norm and matrix norm. The vector norm and matrix norm has different formulation. Therefore how to tell torch.norm to select vector norm or matrix norm. thanks very much.

[JoelMarcey]

@dragen1860 Hi! Just curious how you came up with me to ask your question?

[yaroslavvb]

Related https://github.com/pytorch/pytorch/issues/12760

[yaroslavvb]

@jacobrgardner was suggesting to implement it through Lanczos which has a reference implementation in https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/utils/lanczos.py

[mruberry]

@kurtamohler I think we can close this now, right?

[kurtamohler]

Yep, torch.linalg.norm has numpy-like functionality, including matrix 2-norm.