tvercaut
commented
6 months ago As mentioned in #13, in most practical cases involving sparse matrices, you would want the derivative with respect to the non-zero selements only.
There is some further discussion here: https://github.com/pytorch/pytorch/issues/87448
I haven't checked in a while but at the time, pytorch had some issues with gradcheck and sparse matrix operations:
https://github.com/pytorch/pytorch/issues/87085
In any case, you probably need to pass check_sparse_nnz=True to gradcheck if you are indeed using the standard sparse ops semantics.
Finally, if you want, we consolidate the pure python functions above here: https://github.com/cai4cai/torchsparsegradutils
zhf-0
Thank you very much for providing such a good tool!
My problem is that when the input
Ais a 'real' sparse matrix, not the sparse matrix converted from a dense matrix, thetorch.autograd.gradcheck()function will throw an exception. Thepythonprogram I use iswhich is based on the program from Differentiable sparse linear solver with cupy backend - “unsupported tensor layout: Sparse” in gradcheck, whose author @tvercaut wrote the program based on your blog and program. I modified the program and limited it to running only on
CPU.The output is
The success of your and @tvercaut's program in passing the gradient check can be attributed to the fact that the sparse matrix
Ayou used is actually a dense matrix. Consequently, theautograd()function computes the gradient for each element.The derivative formula from your blog is $$\frac{\partial L}{\partial A} = - \frac{\partial L}{\partial b} \otimes x$$ Since the matrix
Ais sparse, then $\frac{\partial L}{\partial A{ij}}=0$ when $A{ij}=0$, but the results computed bypytorchshow it's not true. If I changebackward()function intoThen the gradient check is passed. However the
gradAis now a dense matrix, which is not consistent to the theoretical result. There is a similar issue #13 without detailed explanation. So I want to ask which gradient is right ? the sparse one or the dense one?