search

barkincavdaroglu / Link-Prediction-Mesh-Network

PyTorch Implementation of a Deep Learning Model for Temporal Link Prediction in MANETs
2 stars 1 forks source link

Pytorch Lightning gradient clipping doesn't work when manually optimizing #14

Closed barkincavdaroglu closed 1 year ago

barkincavdaroglu commented 1 year ago

Since Lightning doesn't yet allow automatic optimization with multiple optimizers, I had to switch to manual optimization, which doesn't allow for gradient clipping.

torch.nn.utils.clip_grad_value_(
    model.parameters(), self.gradient_clip_val
)

When we use the code above before stepping and after calling backward on the loss, there is a significant overhead that slows down the training.

barkincavdaroglu commented 1 year ago

Removed GAN and enabled automatic_optimization. So there is no need for manually clipping grads.