search

pytorch / PiPPy

Pipeline Parallelism for PyTorch
BSD 3-Clause "New" or "Revised" License
664 stars 79 forks source link

Can Pippy be combined with PEFT LoRA? #1122

Open Songjw133 opened 4 weeks ago

Songjw133 commented 4 weeks ago

I'm not very familiar with pipeline parallelism. Can it work if most of the model's parameters are frozen?

kwen2501 commented 3 weeks ago

Hi, good question, I haven't tried it myself and don't have much experience with PEFT. Do you have a use case in hand? For the forward pass, it should still work if you provide the PEFT'ed model to PP's API. For the backward pass, we rely on an assumption that the backward flow of gradients have the same size as the forward flow of activations. Do you think this assumption still holds in PEFT case?