search

cloneofsimo / paint-with-words-sd

Implementation of Paint-with-words with Stable Diffusion : method from eDiff-I that let you generate image from text-labeled segmentation map.
MIT License
636 stars 52 forks source link

Switch to SD v1.5 #12

Closed 143f86b6-91b7-47e4-808b-18779e22f2dd closed 2 years ago

143f86b6-91b7-47e4-808b-18779e22f2dd commented 2 years ago

Is it possible to use this in v1.5 instead of v1.4?

0xdevalias commented 2 years ago

This is documented in the README already:

You can see that pww_load_tools takes a hf_model_path and/or local_model_path here:

https://github.com/cloneofsimo/paint-with-words-sd/blob/4c95a11a5d8c9d691d942163667bcb6dba944daf/paint_with_words/paint_with_words.py#L89-L93

pww_load_tools is called by paint_with_words, which can also be passed hf_model_path and/or local_model_path, with hf_model_path defaulting to "CompVis/stable-diffusion-v1-4"

https://github.com/cloneofsimo/paint-with-words-sd/blob/4c95a11a5d8c9d691d942163667bcb6dba944daf/paint_with_words/paint_with_words.py#L213-L238

So it looks like it may be even easier than the README suggests, and you should just be able to pass "runwayml/stable-diffusion-v1-5" directly into the hf_model_path parameter of paint_with_words as it's called in the 'Basic Usage' section:

Something like this would presumably work:

img = paint_with_words(
    color_context=color_context,
    color_map_image=color_map_image,
    input_prompt=input_prompt,
    num_inference_steps=30,
    guidance_scale=7.5,
    device="cuda:0",
+   hf_model_path="runwayml/stable-diffusion-v1-5"
)
0xdevalias commented 2 years ago

I made a simple colab example just now, you can try it out here: paint-with-words colab

Originally posted by @shreydan in https://github.com/cloneofsimo/paint-with-words-sd/issues/4#issuecomment-1312768401

You can confirm the above hf_model_path change on the colab mentioned here by editing the cell that has the img = paint_with_words( (first cell after the 'Run Model' heading)

We can also explicitly set the seed parameter here as well, to make comparisons between the 2 models easier.

Original version using SD v1.4

image image

New version using SD v1.5

image image

hiss-remi commented 2 years ago

You can see that pww_load_tools takes a hf_model_path and/or local_model_path here:

One advantage of this method that you can also change the scheduler, which might be desirable for performance or quality issues. (I found recently that the model I'm using often has blatant graphical issues even with a large step count on LMS, but EulerAncestralDiscreteScheduler or EulerDiscreteScheduler do much better. Just don't forget to import your scheduler of choice from diffusers. (Something the example currently doesn't show.)

cloneofsimo commented 2 years ago

Nice note @hiss-remi , ill add this feature as well.