one of the variables needed for gradient computation has been modified by an inplace operation: which is output 0 of MmBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

xiaoboCASIA / SV-X-Softmax

185 stars 20 forks source link

Open xysong1201 opened 4 years ago

xysong1201 commented 4 years ago

Hi, I tried your MV-AM and MV-Arc , both give me this error. Someone encountered the same error?

BigTom007 commented 3 years ago

I meet the same problem when I train the code on multi server, but I don't know the reason

BigTom007 commented 3 years ago

I solve the problem, Multi Sever can not use mask operation, you can use torch.where to replace it

cos_theta_hard = (self.t + 1.0) * cos_theta + self.t cos_theta = torch.where(cos_theta > gt -self.margin, cos_theta_hard, cos_theta)