search

wenquanlu / HandRefiner

MIT License
701 stars 30 forks source link

side effects of handrefiner #12

Open WangYuXuan2022 opened 5 months ago

WangYuXuan2022 commented 5 months ago

Hi, I have tested several images generated by myself on stable diffusion webui. It found out although the corrupted hands have been rectified, some parts like face and mouth appear slight distorted which makes the generated image look unnatural. Did you meet such situations before, if so, how can I fix it, thank you!

wenquanlu commented 5 months ago

Hi, this concerns a broader issue of stable diffusion's inpainting, as in its original implementation, the unmasked area may be slightly changed. Please see this issue https://github.com/huggingface/diffusers/issues/3880 for more details.

One solution is to force the unmasked area unchanged, however, there may be inconsistencies in the transition to the masked area:

import cv2

mask = cv2.imread("mask.jpg")
original = cv2.imread("original_image.jpg")
generated = cv2.imread("generated_image.jpg")

def unchange_unmasked(mask, original, generated):
    mask = mask/255
    mask = 1 * (mask > 0.5)
    return (1 - mask) * original + mask * generated

new = unchange_unmasked(mask, original, generated)
cv2.imwrite("new.jpg",new)

In automatic1111, I guess they apply some techniques to smooth out the transition, I am not too sure about the detail. For anyone who browsed over this post and knew any effective solution, please feel free to share your knowledge and insights, thanks!

xiankgx commented 3 months ago

The changing of unmasked areas is caused by VAE imperfect reconstruction of the input.