Open YangYongJin opened 3 months ago
How do you find this problem? Can you tell me the right accelerate version? Thank you very much!
Can updating the accelerator to the latest version resolve this issue?
Yes, in my case, just updating the accelerator to the latest version resolve this issue!
Hi, I believe there is a significant issue with the Hugging Face Accelerate version. This code is using version 4.20, according to the
requirements.txt
, but that version has a serious bug in theaccelerator.accumulate
function. After the first epoch, gradients are not accumulated at all, rendering the performance after the first epoch meaningless (approximately 400 steps in this case). I suggest upgrading the Accelerate version as the recent version does not have this issue.