Closed odusseys closed 6 months ago
Got it working here: https://colab.research.google.com/gist/sayakpaul/0197712664287f2548844c7e90edf298/scratchpad.ipynb.
Main fix:
unet.load_state_dict(load_file(hf_hub_download(model_repo, model_ckpt)))
+ unet.to(dtype=torch.float16)
Describe the bug
I am running SDXL-lightning with a canny edge controlnet. Running on a T4 (16G VRAM). I am using
enable_model_cpu_offload
to reduce memory usage, but I am running into the following error:mat1 and mat2 must have the same dtype
I am guessing something is wrong with the offloading of some components, leading to incompatible dtypes in weights and inputs. I am only using PIL images as inputs, and thus would expect the pipeline to handle type conversions.
Reproduction
Logs
System Info
diffusers
version: 0.27.0.dev0Who can help?
@yiyixuxu @sayakpaul @DN6