Open ememos opened 5 months ago
activation_ckpt
, force_free_mem
, swap_opt_in_fwdbwd
, and swap_model_in_optstep
. These options belong to the Optimus_p
class and can be set to either True or False.ir_analyze
argument of the Optimus_p
class ( IR_Anal.PARALLEL
or IR_Anal.SEQUENTIAL
).cache_dir
option of from_pretrained
in combination with dist.barrier
to ensure that only one process downloads the Hugging Face model, while other processes utilize the cache. This provides an application-level workaround.
We need the ability to train models that exceed the size of GPU memory