search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
17.07k stars 1.96k forks source link

[Bug]: Loopback + inpainting #2090

Closed sangoi-exe closed 12 months ago

sangoi-exe commented 1 year ago

Is there an existing issue for this?

What happened?

Hello! I'm not sure if the bug is in the CN, in the A1111, or in the Loopback script, but when I try to use loopback with the CN for inpainting, with the aim of not having to use a specific inpainting model (I'm getting runtime errors with any inpainting model, but that's beside the point), the result ends up as if I'm not using the CN and am using a regular model for inpainting instead, leaving the masked area with a bizarre color.

Below I've attached my settings and an example with loopback as well as one without loopback.

001 002 003 004

Loopback + ControlNet tmphrf5l7cz

4x times with only ControlNet, sending the image to input manually: 00037-2621282885 00038-4164855227 00039-3597955738 00040-3021731623

Steps to reproduce the problem

To reproduce, simply use loopback with a regular model and the ControlNet for inpaint.

What should have happened?

An picture without the blue weird mark.

Commit where the problem happens

webui: 5ef669de080814067961f28357256e8fe27544f4 controlnet: v1.1.408

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

Just --xformers

List of enabled extensions

Built-in + sd-webui-3d-open-pose-editor sd-webui-controlnet stable-diffusion-webui-wd14-tagger ultimate-upscale-for-automatic1111

Console logs

ControlNet + Loopback + Normal SD model

2023-09-09 12:53:49,792 - ControlNet - INFO - Loading model: control_v11p_sd15_inpaint [ebff9138]
2023-09-09 12:53:50,838 - ControlNet - INFO - Loaded state_dict from [C:\stable-diffusion-webui\models\ControlNet\control_v11p_sd15_inpaint.pth]
2023-09-09 12:53:50,838 - ControlNet - INFO - controlnet_default_config
2023-09-09 12:53:52,494 - ControlNet - INFO - ControlNet model control_v11p_sd15_inpaint [ebff9138] loaded.
2023-09-09 12:53:52,584 - ControlNet - INFO - Loading preprocessor: inpaint_only+lama
2023-09-09 12:53:52,584 - ControlNet - INFO - preprocessor resolution = -1
2023-09-09 12:53:54,813 - ControlNet - INFO - ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 64, 64]).
2023-09-09 12:53:54,875 - ControlNet - INFO - ControlNet Hooked - Time = 5.290883302688599
2023-09-09 12:53:55,134 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
  0%|                                                                          | 0/3 [00:00<?, ?it/s]2023-09-09 12:53:55,143 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 33%|██████████████████████                                            | 1/3 [00:00<00:01,  1.42it/s]2023-09-09 12:53:55,845 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 67%|████████████████████████████████████████████                      | 2/3 [00:00<00:00,  2.30it/s]2023-09-09 12:53:56,092 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
100%|██████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00,  2.23it/s]
2023-09-09 12:53:57,278 - ControlNet - INFO - Loading model from cache: control_v11p_sd15_inpaint [ebff9138]
2023-09-09 12:53:57,292 - ControlNet - INFO - Loading preprocessor: inpaint_only+lama
2023-09-09 12:53:57,292 - ControlNet - INFO - preprocessor resolution = -1
2023-09-09 12:53:57,601 - ControlNet - INFO - ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 64, 64]).
2023-09-09 12:53:57,619 - ControlNet - INFO - ControlNet Hooked - Time = 0.3405461311340332
2023-09-09 12:53:57,823 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
  0%|                                                                          | 0/3 [00:00<?, ?it/s]2023-09-09 12:53:57,825 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 33%|██████████████████████                                            | 1/3 [00:00<00:00,  2.43it/s]2023-09-09 12:53:58,235 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 67%|████████████████████████████████████████████                      | 2/3 [00:00<00:00,  3.32it/s]2023-09-09 12:53:58,460 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
100%|██████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00,  2.97it/s]
2023-09-09 12:53:59,597 - ControlNet - INFO - Loading model from cache: control_v11p_sd15_inpaint [ebff9138]
2023-09-09 12:53:59,611 - ControlNet - INFO - Loading preprocessor: inpaint_only+lama
2023-09-09 12:53:59,612 - ControlNet - INFO - preprocessor resolution = -1
2023-09-09 12:53:59,914 - ControlNet - INFO - ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 64, 64]).
2023-09-09 12:53:59,930 - ControlNet - INFO - ControlNet Hooked - Time = 0.3325486183166504
2023-09-09 12:54:00,133 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
  0%|                                                                          | 0/4 [00:00<?, ?it/s]2023-09-09 12:54:00,135 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 25%|████████████████▌                                                 | 1/4 [00:00<00:01,  2.47it/s]2023-09-09 12:54:00,540 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 50%|█████████████████████████████████                                 | 2/4 [00:00<00:00,  3.35it/s]2023-09-09 12:54:00,765 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 75%|█████████████████████████████████████████████████▌                | 3/4 [00:00<00:00,  3.04it/s]2023-09-09 12:54:01,130 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
100%|██████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00,  3.33it/s]
2023-09-09 12:54:02,072 - ControlNet - INFO - Loading model from cache: control_v11p_sd15_inpaint [ebff9138]
2023-09-09 12:54:02,086 - ControlNet - INFO - Loading preprocessor: inpaint_only+lama
2023-09-09 12:54:02,086 - ControlNet - INFO - preprocessor resolution = -1
2023-09-09 12:54:02,393 - ControlNet - INFO - ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 64, 64]).
2023-09-09 12:54:02,410 - ControlNet - INFO - ControlNet Hooked - Time = 0.3381531238555908
2023-09-09 12:54:02,612 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
  0%|                                                                          | 0/6 [00:00<?, ?it/s]2023-09-09 12:54:02,614 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 17%|███████████                                                       | 1/6 [00:00<00:02,  2.44it/s]2023-09-09 12:54:03,023 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 33%|██████████████████████                                            | 2/6 [00:00<00:01,  3.33it/s]2023-09-09 12:54:03,247 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 50%|█████████████████████████████████                                 | 3/6 [00:01<00:01,  2.99it/s]2023-09-09 12:54:03,621 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 67%|████████████████████████████████████████████                      | 4/6 [00:01<00:00,  3.53it/s]2023-09-09 12:54:03,827 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
 83%|███████████████████████████████████████████████████████           | 5/6 [00:01<00:00,  3.28it/s]2023-09-09 12:54:04,172 - ControlNet - INFO - [ControlNet] Initial noise hack applied to torch.Size([1, 4, 64, 64]).
100%|██████████████████████████████████████████████████████████████████| 6/6 [00:01<00:00,  3.43it/s]
Total progress: 16it [00:09,  1.73it/s]
{"prompt": "dslr, ultra quality, sharp focus, tack sharp, Fujifilm XT3", "all_prompts": ["dslr, ultra quality, sharp focus, tack sharp, Fujifilm XT3"], "negative_prompt": "(blurry un-sharp fuzzy un-detailed skin:1.4), (semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, doll, overexposed, photoshop, oversaturated)", "all_negative_prompts": ["(blurry un-sharp fuzzy un-detailed skin:1.4), (semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, doll, overexposed, photoshop, oversaturated)"], "seed": 2661430128, "all_seeds": [2661430131], "subseed": -1, "all_subseeds": [3978185627], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler", "cfg_scale": 12, "steps": 40, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "icbinpICantBelieveIts_seco", "sd_model_hash": "fa1224c923", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.19999999999999996, "extra_generation_params": {"Final denoising strength": 0.15, "Denoising curve": "Lazy", "ControlNet 0": "Module: inpaint_only+lama, Model: control_v11p_sd15_inpaint [ebff9138], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: My prompt is more important"}, "index_of_first_image": 0, "infotexts": ["dslr, ultra quality, sharp focus, tack sharp, Fujifilm XT3\nNegative prompt: (blurry un-sharp fuzzy un-detailed skin:1.4), (semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, doll, overexposed, photoshop, oversaturated)\nSteps: 40, Sampler: Euler, CFG scale: 12, Seed: 2661430128, Size: 512x512, Model hash: fa1224c923, Model: icbinpICantBelieveIts_seco, Denoising strength: 0.05, NGMS: 1, Final denoising strength: 0.15, Denoising curve: Lazy, ControlNet 0: \"Module: inpaint_only+lama, Model: control_v11p_sd15_inpaint [ebff9138], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: My prompt is more important\", Version: v1.6.0"], "styles": [], "job_timestamp": "20230909125349", "clip_skip": 1, "is_using_inpainting_conditioning": false}

Additional information

Sorry if this isn't ControlNet bug.

lhucklen commented 1 year ago

I am having same issue