search

lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
29.76k stars 2.69k forks source link

just a finding of generating human faces or half-body image using canny, hed, etc... #180

Open zihaog0724 opened 1 year ago

zihaog0724 commented 1 year ago

Hi,

Firstly i would like to say many thanks to the developers of ControlNet. Great work!

I've been using ControlNet to generate human-related stuff, e.g. faces, half-body portrait, etc... One interesting issue i found would be the model tends to generate grayscale (white and black) image when canny/hed conditions is being added and idk why. I have to say this phenomenon is indeed different from other stable diffusion models that i've used.

One feasible solution to alleviate this issue is simply add "grayscale, monochrome" to the negative prompt if you don't want grayscale effects.

lllyasviel commented 1 year ago

webui plugin does not have this problem. not sure what happens.

zihaog0724 commented 1 year ago

@lllyasviel I did not use webui plugin. I just simply used the process function in gradio_canny2image.py and save results in opencv. Human faces did show black and white effects. Once I changed cat, dog, etc from human without modifying any code, the issue has gone.

lllyasviel commented 1 year ago

probably caused by that ddim. will double check

zihaog0724 commented 1 year ago

@lllyasviel ok thanks! If any root cause is found and solved plz let me know

geroldmeisinger commented 12 months ago

this may actually have been an issue in 1.0 which is fixed in 1.1, see here https://github.com/lllyasviel/ControlNet-v1-1-nightly

The training dataset of previous cnet 1.0 has several problems including (1) a small group of greyscale human images are duplicated thousands of times (!!), causing the previous model somewhat likely to generate grayscale human images; (2) some images has low quality, very blurry, or significant JPEG artifacts; (3) a small group of images has wrong paired prompts caused by a mistake in our data processing scripts. The new model fixed all problems of the training dataset and should be more reasonable in many cases.