The roles of the aforementioned options are confusing, and it would be beneficial to change them to more clearly defined meanings. The optimizer_offload option sends the optimizer state to the CPU when GPU memory runs low during the forward/backward passes, and swaps it back to the GPU during other phases. The model_offload option swaps parameters, gradients, and optimizer states to the CPU during the optimizer step() phase, and swaps parameters and gradients back to the GPU during the forward/backward passes for training.
The optimizer_offload option has been changed to the swap_opt_in_fwdbwd option. Additionally, the model_offload option has been changed to the swap_model_in_optstep option.
The roles of the aforementioned options are confusing, and it would be beneficial to change them to more clearly defined meanings. The optimizer_offload option sends the optimizer state to the CPU when GPU memory runs low during the forward/backward passes, and swaps it back to the GPU during other phases. The model_offload option swaps parameters, gradients, and optimizer states to the CPU during the optimizer step() phase, and swaps parameters and gradients back to the GPU during the forward/backward passes for training.