Closed egillax closed 3 months ago
By summing the gradients for a number of accumulation_steps before calling backward we can use lower batch sizes to reduce memory when otherwise it would run out of memory.
accumulation_steps
backward
This should be pretty trivial to implement.
Fixed by #113
egillax
commented
3 months ago
By summing the gradients for a number of
accumulation_stepsbefore callingbackwardwe can use lower batch sizes to reduce memory when otherwise it would run out of memory.This should be pretty trivial to implement.