search

liuliu / swift-diffusion

BSD 3-Clause "New" or "Revised" License
429 stars 33 forks source link

How to load weights into model? #55

Closed davidw0311 closed 8 months ago

davidw0311 commented 9 months ago

Hello, I am having difficulty trying to run txt2img on my macbook pro M1. I downloaded the sd-v1.4.ckpt from the link in the readme, but run into the same issue as https://github.com/liuliu/swift-diffusion/issues/49

In trying to debug this, when I print all the keys from the checkpoint, I notice they are labeled as:

image

So I am wondering if image

is the correct way to load the weights into the model (and similarly for the text model and encoder/decoder?

However, I am still getting nan values as the output from my decoder at the end, wondering if this is related to the weight loading or something else?

Thank you!

limengf1218 commented 9 months ago

Loading with graph.openStore("path_to_sd1.5.ckpt") { store in store.read("text_model", model: textModel) }

is working, but the problem seems to be using the LoRAUnet or LoRACLIPTextModel. Loading with UNet() and CLIPTextModel instead works