lucasjinreal / DCNv2_latest

DCNv2 supports decent pytorch such as torch 1.5+ (now 1.8+)
BSD 3-Clause "New" or "Revised" License
637 stars 124 forks source link

Jacobian mismatch for output ? #45

Open K-M-Ibrahim-Khalilullah opened 3 years ago

K-M-Ibrahim-Khalilullah commented 3 years ago

Thanks for the latest repo. Sometimes it produce this error:

UserWarning: The {}th input requires gradient and is not a double precision floating point or complex. This check will likely fail if all the inputs are not of double precision floating point or complex. 'The {}th input requires gradient and ' check_gradient_dpooling: True Traceback (most recent call last): File "testcuda.py", line 265, in check_gradient_dconv() File "testcuda.py", line 97, in check_gradient_dconv eps=1e-3, atol=1e-4, rtol=1e-2)) File "\lib\site-packages\torch\autograd\gradcheck.py", line 390, in gradcheck checkIfNumericalAnalyticAreClose(a, n, j) File "lib\site-packages\torch\autograd\gradcheck.py", line 372, in checkIfNumericalAnalyticAreClose 'numerical:%s\nanalytical:%s\n' % (i, j, n, a)) File "\lib\site-packages\torch\autograd\gradcheck.py", line 289, in fail_test raise RuntimeError(msg) RuntimeError: Jacobian mismatch for output 0 with respect to input 1, numerical:tensor([[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0001, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], ..., [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]], device='cuda:0') analytical:tensor([[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0001, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], ..., [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000], [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]], device='cuda:0')

Is there any solution?

ethem-kinginthenorth commented 3 years ago

I am having the same issue. It randomly complains on both cpu and gpu.

lala-tree commented 2 years ago

@AI-ML-Enthusiast have you solved the problem?