search

zewei-Zhang / GoodDrag

Ofiicial GoodDrag implementation.
Apache License 2.0
88 stars 6 forks source link

SDXL model #4

Open DAVEISHAN opened 4 months ago

DAVEISHAN commented 4 months ago

Hello, thank you for the awesome work! I have tried changing the diffusion model to SDXL and it produces some problematic output, can you pl look into this? I am attaching the example here.

image download (9)

zewei-Zhang commented 4 months ago

Hi, thank you for your interest in our work! Currently, SDXL is not supported as its pipeline involves two diffusion models and differs significantly from SD 1.5 or SD 2. Are you experiencing any issues running on SD 1.5 or SD 2?

DAVEISHAN commented 4 months ago

Hi, thank you for your clarification. I have tried SD1.5, which works fine for me with the demo images as shown in your paper. However, I am facing a problem where it loses fine-grained details such as facial features and textures, and generates distorted images. For example, see the image below. Is there a way to resolve this issue? Input Image (512 x 512): 2022-06-11-183524_5ave_39st-40st Drag:

image

Output image: download (14)

Also, any insights on the integration of the SDXL would be very helpful.

zewei-Zhang commented 4 months ago

This is due to the limitations of SD1.5 /2 + Lora in image inversion. I tried with the same image without performing any drag operations, just inversion, and got similar results to yours. For handling images with a lot of detail, I recommend try some diffusion inversion papers. Ensure the quality of inversion before proceeding with any drag operations. Compared to SD1.5/2, SDXL is more sensitive. If you need to use SDXL, it might require experimenting with different parameters on specific images. Additionally, I suggest incorporating the latter half of the SDXL diffusion model to enhance the results.