Open ememos opened 2 months ago
An OOM error might occur in GPU memory during the execution of optimizer.step()
The model_offload option added. With this option, during the optimizer.step() interval, parameters, gradients, and optimizer states are all offloaded to the CPU, and the optimizer step is performed on the CPU.
An OOM error might occur in GPU memory during the execution of optimizer.step()