Right now, step is reset across epochs. This is problematic if, say, you're training on a dataset with 8 examples but have a per-gpu batch size of 8 and want to do gradient accumulation for 4 steps. You'll never reach step = 4, and then never do gradient updates.
Right now,
step
is reset across epochs. This is problematic if, say, you're training on a dataset with 8 examples but have a per-gpu batch size of 8 and want to do gradient accumulation for 4 steps. You'll never reachstep = 4
, and then never do gradient updates.this fix just persists
step
across epochs