Closed pumplerod closed 8 months ago
Hi @pumplerod, thank you for your patience. I was able to successfully reproduce the nan
gradients with you code above. It does seem like reducing increasing the regularization strength does fix the issue, but I will look into this more closely, as ideally the gradients should always be defined, even if they are just 0.
Perhaps I am missing something obvious however I found
kl
regularization to work a bit better for my data so I switch froml2
tokl
and began to getnan
values in the gradient.Here is the test code I used to verify...
Python 3.8.10
torch==1.13.0
torchsort==0.1.9
Use of
l2
works as expected.Use of
kl
returnsnan
.