search

BlenderNeko / ComfyUI_Cutoff

cutoff implementation for ComfyUI
GNU General Public License v3.0
366 stars 21 forks source link

Issue with Realistic Models #3

Open paulo-coronado opened 1 year ago

paulo-coronado commented 1 year ago

Hello! I am trying to apply Cutoff to realistic models, such as RealisticVision2.0 and the tokens doesn't work as expected. Any ideas? I think it is because the model was not trained with this colored regions, so it refuses to generate as requested.

This is an image generated with the following prompt:

beautiful european model, white shirt with green tie, black shoes, blue hair, yellow eyes, pink skirt, full body shot, walking on a street

image

BlenderNeko commented 1 year ago

Unfortunately your mileage may vary depending on the model that you're using. After playing around with realistic vision for a bit, it seems that realistic vision appears particularly bad for this, at least when it comes to colors.

arcusmaximus commented 1 year ago

For A1111 there's the Regional Prompter extension whose region specification by prompt feature can be used for the same goal (preventing color bleeding) but with a completely different approach. Given a prompt like

beautiful european model, tie BREAK
green, tie

it can find the area in the image that corresponds to the tie keyword and apply the green prompt just to that. It also previews the masks that it found, and lets you grow/shrink them by tweaking their threshold values.

Here's a quick one I did with, yes, RealisticVision 2.0:

tmpd_uxlzoo

So this approach seems more robust than patching embedding vectors - case in point, A1111 Cutoff gave me rather mixed results even for anime. Having looked at ComfyUI for the first time today, though, I'm not sure something similar exists there yet.