Failed to train large models that exceed the aggregated size of all GPU memories

[ememos]

We need the ability to train models that exceed the size of GPU memory

[ememos]

• We are adding features to address memory shortage issues. The following options have been added to deal with GPU memory shortages: activation_ckpt, force_free_mem, swap_opt_in_fwdbwd, and swap_model_in_optstep. These options belong to the Optimus_p class and can be set to either True or False.
• Regarding CPU memory shortages, you can set the ir_analyze argument of the Optimus_p class ( IR_Anal.PARALLEL or IR_Anal.SEQUENTIAL).
• Additionally, CPU OOM (Out of Memory) issues can occur when downloading Hugging Face model parameters. In such cases, you can use the cache_dir option of from_pretrained in combination with dist.barrier to ensure that only one process downloads the Hugging Face model, while other processes utilize the cache. This provides an application-level workaround.