[Major Update] Detail transfer / Reinforce FG and more

huchenlei commented 3 months ago

This PR is based on #23.

@Haoming02 did amazing improvement on the IC-Light extension.

Here is the list of actual changes made in Haoming's fork:

Reorganize UIs
Add explanations on how each mode works
Load models from ic-light folder with arbitrary filenames
Use pre-built rembg package instead
Important: Swap the way how img2img works: Now the input image is the subject while the extension takes lighting conditions.

This just makes more sense to me...
New: Implement Difference of Gaussians to reintroduce some details after the processing

My changes made on top of Haoming's fork:

Revert the change that swaps input_fg and img2img input. img2img input can be used by other extensions such as ControlNet. We should not swap it mid way, which can cause confusion.
Moved some scripts from /script folder to /libiclight folder. The general guideline is that you don't want to put a file with generic name in scripts folder as the namespace is shared with other extensions.
New: Add reinforce fg checkbox which can help preserve the base color of fg object. This option essentially does the same processing as https://github.com/huchenlei/ComfyUI-IC-Light-Native/blob/main/examples/ic_light_preserve_color.json

Breaking change:

You need to move IC-Light models from models/unet to models/ic-light

Updated How to use section:

How to use

For best result, it is recommended to use low CFG and strong denosing strength.

Given FG, Generate BG and relight [Txt2Img][HR available]

Infotext:

sunshine from window, a beautiful girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed: 3615474644, Size: 512x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Clip skip: 2, Version: v1.9.3-13-g8e355f08

Given FG and light map, Genereate BG and relight [Img2Img]

Img2Img input image is lightmap. After you select value from the radio, the img2img input will automatically be set by the extension. You can also upload your own lightmap by selecting Custom.

Infotext:

sunshine from window, a beautiful girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed: 2984374687, Size: 512x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Denoising strength: 0.96, Clip skip: 2, Version: v1.9.3-13-g8e355f08

Given FG and BG, Put FG on BG and relight [Txt2Img][HR available]

Infotext:

a beautiful girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed: 2230971215, Size: 512x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Clip skip: 2, Version: v1.9.3-13-g8e355f08

[2024-06-06] Restore Detail

Detail transfer was originally implemented in https://github.com/kijai/ComfyUI-IC-Light. It captures high frequency details, e.g. text, in the input fg image and reapplys them to the output image.

Original output:

After detail restore:

Infotext:

A bottle of oyster sauce, kitchen counter, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed: 2984374687, Size: 768x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Denoising strength: 0.96, Clip skip: 2, Version: v1.9.3-13-g8e355f08

[2024-06-06] Reinforce FG

A big problem of IC-Light is that it often alters the FG object's base color too much. By adding the fg image on top of the lightmap, this issue can be alleviated. This essentially implements this comfyui workflow. Here is a comparison:

Without reinforce-fg: You can observe that the fg object almost looks transparent.

With reinforce-fg (Same seed): The fg object no longer look transparent.

Skip remove bg

If the default remove bg cannot achieve your desired effect, you can use other tools to create an RGBA image and uncheck the remove bg checkbox. Image with grey background can also be used the same way as RGBA image.

Haoming02 commented 3 months ago

img2img input can be used by other extensions such as ControlNet

Hence why doesn't it make more sense when the img2img input is your subject?

What use is sending a black and white gradient to ControlNet anyway?

huchenlei commented 3 months ago

img2img input can be used by other extensions such as ControlNet

Hence why doesn't it make more sense when the img2img input is your subject?

What use is sending a black and white gradient to ControlNet anyway?

If you make people use img2img input as fg input, they might think the fg image will be passed to ControlNet when "use independent control" checkbox is not checked. What they really need to do is check that checkbox and upload fg image again to ControlNet unit for correct control image.

I think swapping the images is just too risky. It also introduces more mental effort when the user compares the ComfyUI impl and Forge/A1111 impl.

Haoming02 commented 3 months ago

the fg image will be passed to ControlNet when "use independent control" checkbox is not checked

Yes. That's my intention. The img2img input, is, well, the input. The lightmaps is used by the Extension, so it belongs to the Extension instead.

When you generate a portrait in txt2img, and click Send to img2img, where does it send the image to? That's right, the img2img input.

Again, this just makes more sense imho

~~And I revert your revert again in my own fork~~

Haoming02 commented 3 months ago

swapping the images is just too risky

Honestly, the only reason why I did the swapping approach is cause I don't know how the original model works. So I just swap them early to continue with your implementaion. If IC Light can take the images directly, then swapping wouldn't be needed.

huchenlei commented 3 months ago

swapping the images is just too risky

Honestly, the only reason why I did the swapping approach is cause I don't know how the original model works. So I just swap them early to continue with your implementaion. If IC Light can take the images directly, then swapping wouldn't be needed.

I think the txt2img/img2img design is the core problem. A1111 does not allow you launch img2img task without an input image, so the lightmap upload cannot be in the extension accordion.

In ComfyUI it is much more clear that txt2img is just setting initial latent as random noise.

huchenlei commented 3 months ago

the fg image will be passed to ControlNet when "use independent control" checkbox is not checked

Yes. That's my intention. The img2img input, is, well, the input. The lightmaps is used by the Extension, so it belongs to the Extension instead.

When you generate a portrait in txt2img, and click Send to img2img, where does it send the image to? That's right, the img2img input.

Again, this just makes more sense imho

~And I revert your revert again in my own fork~

How about we make it a checkbox in the settings so that users can choose which behaviour they want? This should also free you maintaining a separate fork.

Haoming02 commented 3 months ago

Yeah, that would probably be the better idea

changpeidong commented 3 months ago

Installed the latest version of ic-light, the model selection box cannot be opened, the model cannot be selected, the production map is invalid. ![Uploading 屏幕截图 2024-06-26 175645.jpg…]()

huchenlei / sd-forge-ic-light