I saw that:
optimizer_proj.zero_grad() optimizer_distill.zero_grad() loss.backward() if (i + 1) % accumulation_steps == 0: optimizer_proj.step() optimizer_distill.step()
did you mean
loss.backward() if (i + 1) % accumulation_steps == 0: optimizer_proj.step() optimizer_distill.step() optimizer_proj.zero_grad() optimizer_distill.zero_grad()
I saw that:
optimizer_proj.zero_grad() optimizer_distill.zero_grad() loss.backward() if (i + 1) % accumulation_steps == 0: optimizer_proj.step() optimizer_distill.step()
did you meanloss.backward() if (i + 1) % accumulation_steps == 0: optimizer_proj.step() optimizer_distill.step() optimizer_proj.zero_grad() optimizer_distill.zero_grad()