Is it something wrong with grad acc?

[ArlixLin]

I saw that: optimizer_proj.zero_grad() optimizer_distill.zero_grad() loss.backward() if (i + 1) % accumulation_steps == 0: optimizer_proj.step() optimizer_distill.step() did you mean loss.backward() if (i + 1) % accumulation_steps == 0: optimizer_proj.step() optimizer_distill.step() optimizer_proj.zero_grad() optimizer_distill.zero_grad()

[tientrandinh]

Hi, you are right @ArlixLin, I fixed it. Thank you