search

siliconflow / onediff

OneDiff: An out-of-the-box acceleration library for diffusion models.
https://github.com/siliconflow/onediff/wiki
Apache License 2.0
1.56k stars 94 forks source link

CPU Model Offloading #208

Open abi98213 opened 1 year ago

abi98213 commented 1 year ago

Does oneflow support model offloading like pipe.to('cpu') while all the graphs are being loaded ?

abi98213 commented 1 year ago

If anyone can please answer this @doombeaker

strint commented 1 year ago

offloading like pipe.to('cpu') is not currently supported.

We are working on a feature like this( CPU offload), but it will not be ready to use quickly.

Please tell us why you want this feature, and has any tool provided this feature?

abi98213 commented 1 year ago

I need this feature, since I have 8 Gigs GPU and I cannot load multiple models within the GPU. I have to move models to cpu to load any other model. This thing is easily doable with diffusers like this : pipe.enable_model_cpu_offload(). So I was wondering if that is possible with oneflow Thanks

strint commented 1 year ago

This thing is easily doable with diffusers like this : pipe.enable_model_cpu_offload().

This feature is not ready currently. We are thinking about it.

abi98213 commented 1 year ago

How much time this can take ?

abi98213 commented 1 year ago

@strint If you can please answer

shabri-arrahim commented 1 year ago

I'll leave a comment as I need this mechanism too