search

pytorch / PiPPy

Pipeline Parallelism for PyTorch
BSD 3-Clause "New" or "Revised" License
664 stars 79 forks source link

CPU offloading? #1126

Open Xynonners opened 3 weeks ago

Xynonners commented 3 weeks ago

It seems like pipelining could possibly greatly simplify the implementation of a feature such as fairscale's OffloadModel https://fairscale.readthedocs.io/en/latest/deep_dive/offload.html

Is this something that is feasible?

kwen2501 commented 3 weeks ago

Can you elaborate more? Do you mean that one may offload the entire stage after its forward pass? (And meanwhile bring back another stage from CPU to do the next forward?)

Xynonners commented 3 weeks ago

Can you elaborate more? Do you mean that one may offload the entire stage after its forward pass? (And meanwhile bring back another stage from CPU to do the next forward?)

yes, that is what I was thinking