search

microsoft / DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
https://www.deepspeed.ai/
Apache License 2.0
33.63k stars 3.95k forks source link

Reset Optimizer #5615

Closed ahorazahedi closed 3 weeks ago

ahorazahedi commented 3 weeks ago

I am looking for Way to reset optimizer for relora in native pytorch code i am using sth like this to reset optimizer for lora items for name, param in model.named_parameters(): if 'lora' in name: del optimizer.state[param] but i cant do this for Zero optimizer of deep speed , is there any way for this ?

jomayeri commented 3 weeks ago

I believe you could, the ZeRO optimizer keeps the param groups the same as the torch optimizer. Look in deepspeed/runtime/zero for the attributes you want to reset.