search

cubiq / ComfyUI_InstantID

Apache License 2.0
945 stars 50 forks source link

InstantID masking is not properly applied when used alongside differential diffusion #90

Open Mithrillion opened 3 months ago

Mithrillion commented 3 months ago

Screenshot: image

Issue description: When using masking in both InstantID and masked differential diffusion, the noisy diffusion output from regions outside of masked areas appear to be pasted over the original image.

Expected behaviour: Regions outside of masked areas should not be altered

Observed behaviour: In the last 1-2 steps of sampling (see the advanced ksampler example above), the entire unmasked region is altered

Notes: InstantID appears to work fine with regular inpainting without the differential diffusion node. It also still works if the mask is disconnected from InstantID node - this results in similar output and can be a workaround for now.

cubiq commented 3 months ago

instantid adds a strong conditioning mask, you need to combine (at least in the positive) another conditioning that covers the area not touched by instantid. It can be on the full image or masked to the inverted instantid mask.

Mithrillion commented 3 months ago

I see, the issue should come from areas outside the mask not being conditioned. However I don't quite understand why it only happens with differential diffusion inpainting but not regular inpainting. Both approaches should keep areas far from the mask completely unchanged. What's even more puzzling is that you can see the area outside of the mask being completely fine in the intermediate step up to 28, but it gets destroyed in a single step at 29-30. Maybe some unintended interaction with differential diffusion?