Feature/gradient clip - Githubissues

MathGaron / pytorch_toolbox

Boiler plate code for pytorch. Train/Validation loops, visualization etc. For research.

MIT License

10 stars 3 forks source link

Feature/gradient clip #19

Closed MathGaron closed 6 years ago

MathGaron commented 7 years ago

Might be useful, especially for those of you who use LSTMs...

jacenfox commented 7 years ago

I don't get why the max_norm is set to 1. L139 torch.nn.utils.clip_grad_norm(self.model.parameters(), 1)

MathGaron commented 7 years ago

People usually use 1, (if it gets higher than that you have the exploding gradient problem!) mmh maybe for some unusual case people could need to clip it lower than 1.. but for now I guess that it should be fine?

MathGaron commented 6 years ago

Bump!?