Closed egillax closed 3 months ago
By summing the gradients for a number of accumulation_steps before calling backward we can use lower batch sizes to reduce memory when otherwise it would run out of memory.
accumulation_steps
backward
This should be pretty trivial to implement.
Fixed by #113
By summing the gradients for a number of
accumulation_steps
before callingbackward
we can use lower batch sizes to reduce memory when otherwise it would run out of memory.This should be pretty trivial to implement.