search

facebookresearch / optimizers

For optimization algorithm research and development.
Other
255 stars 25 forks source link

Using Shampoo with Accelerate and FSDP #24

Open kfirgoldberg opened 3 weeks ago

kfirgoldberg commented 3 weeks ago

Hi, Does the Shampoo implementation support HuggingFace's Accelerate library? Can it be used in: model, optimizer, scheduler = accelerator.prepare(model, optimizer, scheduler) ?

Thanks!

tsunghsienlee commented 1 week ago

Hi @kfirgoldberg ,

We haven't tried that with HuggingFace's Accelerate library? I wonder have you tried that, and did you encounter any issue?

kfirgoldberg commented 4 days ago

I didn't try it yet, I'll update if I get to it in the future