Right now, step is reset across epochs. This is problematic if, say, you're training on a dataset with 8 examples but have a per-gpu batch size of 8 and want to do gradient accumulation for 4 steps. You'll never reach step = 4, and then never do gradient updates.
timoschick
commented
3 years ago
Right now,
stepis reset across epochs. This is problematic if, say, you're training on a dataset with 8 examples but have a per-gpu batch size of 8 and want to do gradient accumulation for 4 steps. You'll never reachstep = 4, and then never do gradient updates.this fix just persists
stepacross epochs