cavalleria / cavaface

face recognition training project(pytorch)
MIT License
456 stars 88 forks source link

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [128, 10572]] #64

Closed Nakupenda-7 closed 3 years ago

Nakupenda-7 commented 3 years ago

optimizer.zero_grad() if USE_APEX: with amp.scale_loss(lossx, optimizer) as scaled_loss: scaled_loss.backward() else: lossx.backward() optimizer.step() RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [128, 10572]], which is output 0 of ClampBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

cavalleria commented 3 years ago

please use my docker env to test