Gradient clipping in the wrong place?

jongwook / onsets-and-frames

A Pytorch implementation of Onsets and Frames (Hawthorne 2018)

MIT License

209 stars 66 forks source link

Gradient clipping in the wrong place? #24

Open KinWaiCheuk opened 3 years ago

KinWaiCheuk commented 3 years ago

In your train.py line 101, you have the gradient clipping after the .backward() and .step() operations.

Shouldn't we put the clip_grad_norm_ in between .backward() and .step()?

jongwook commented 3 years ago

Yeah it definitely should. I'm not sure why I did like that. In practice the gradient was well below the clip value during most of the training though.