search

cloneofsimo / paint-with-words-sd

Implementation of Paint-with-words with Stable Diffusion : method from eDiff-I that let you generate image from text-labeled segmentation map.
MIT License
636 stars 50 forks source link

Paint with words with Lora? #45

Open alexblattner opened 1 year ago

alexblattner commented 1 year ago

Hi, I have been wondering if it was possible to use the concept of paint with words for Lora concepts considering how much better they are than textual inversion for teaching a model specific concepts. The issue is how different TIs and Loras are in how they work. TIs just add themselves to the dictionary of the model and Loras essentially modify to whole model itself. So I was wondering if it's possible to apply loras to specific areas of an image considering how loras work.

Is this something you happen to know about @lwchen6309 ? thank in advance

lwchen6309 commented 1 year ago

It's not likely seems to me as the Lora updates the weight parameters instead of the feature map. Yet, you can always confirm this with Simo as he maintained both repos.

alexblattner commented 1 year ago

That's too bad, I hope it is possible