search

glucauze / sd-webui-faceswaplab

Extended faceswap extension for StableDiffusion web-ui with multiple faceswaps, inpainting, checkpoints, ....
https://glucauze.github.io/sd-webui-faceswaplab/
GNU Affero General Public License v3.0
715 stars 92 forks source link

Masking artifacts (small white squares) when using improved segmented mask #38

Closed Crux1y closed 11 months ago

Crux1y commented 11 months ago

Recently my images have been generating masking artifacts around the face. These appear as random, small, overexposed squares around the face in the final image. After some troubleshooting, I found that the only way I can resolve the issue is by disabling "Default Use improved segmented mask (use pastenet to mask only the face) (requires restart)" in the settings.

I'm not sure the exact version, but the issue started for me on 8/6/2023. Issue happens with and without any post-processing.

Desktop (please complete the following information):

A1111 Info:

Additional context 2023-08-07 15:02:45,535 - FaceSwapLab - INFO - Try to use model : C:\A1111\stable-diffusion-webui\models\faceswaplab\inswapper_128.onnx 2023-08-07 15:02:45,540 - FaceSwapLab - INFO - Load analysis model det_size=(640, 640), det_thresh=0.5, providers = ['CUDAExecutionProvider'], will take some time. (> 30s) Loading (640, 640) analysis model (first time is slow): 100%|█████████████████████████| 1/1 [00:00<00:00, 1.04model/s] 2023-08-07 15:02:46,502 - FaceSwapLab - INFO - ("Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with " "options: {'CUDAExecutionProvider': {'do_copy_in_default_stream': '1', " "'cudnn_conv_algo_search': 'EXHAUSTIVE', 'device_id': '0', " "'gpu_external_alloc': '0', 'enable_cuda_graph': '0', 'gpu_mem_limit': " "'18446744073709551615', 'gpu_external_free': '0', " "'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', " "'cudnn_conv_use_max_workspace': '1', 'cudnn_conv1d_pad_to_nc1d': '0', " "'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', " "'enable_skip_layer_norm_strict_mode': '0'}, 'CPUExecutionProvider': {}}\n" 'find model: ' 'C:\A1111\stable-diffusion-webui\models\faceswaplab\analysers\models\buffalo_l\1k3d68.onnx ' "landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0\n" "Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with " "options: {'CUDAExecutionProvider': {'do_copy_in_default_stream': '1', " "'cudnn_conv_algo_search': 'EXHAUSTIVE', 'device_id': '0', " "'gpu_external_alloc': '0', 'enable_cuda_graph': '0', 'gpu_mem_limit': " "'18446744073709551615', 'gpu_external_free': '0', " "'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', " "'cudnn_conv_use_max_workspace': '1', 'cudnn_conv1d_pad_to_nc1d': '0', " "'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', " "'enable_skip_layer_norm_strict_mode': '0'}, 'CPUExecutionProvider': {}}\n" 'find model: ' 'C:\A1111\stable-diffusion-webui\models\faceswaplab\analysers\models\buffalo_l\2d106det.onnx ' "landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0\n" "Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with " "options: {'CUDAExecutionProvider': {'do_copy_in_default_stream': '1', " "'cudnn_conv_algo_search': 'EXHAUSTIVE', 'device_id': '0', " "'gpu_external_alloc': '0', 'enable_cuda_graph': '0', 'gpu_mem_limit': " "'18446744073709551615', 'gpu_external_free': '0', " "'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', " "'cudnn_conv_use_max_workspace': '1', 'cudnn_conv1d_pad_to_nc1d': '0', " "'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', " "'enable_skip_layer_norm_strict_mode': '0'}, 'CPUExecutionProvider': {}}\n" 'find model: ' 'C:\A1111\stable-diffusion-webui\models\faceswaplab\analysers\models\buffalo_l\det_10g.onnx ' "detection [1, 3, '?', '?'] 127.5 128.0\n" "Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with " "options: {'CUDAExecutionProvider': {'do_copy_in_default_stream': '1', " "'cudnn_conv_algo_search': 'EXHAUSTIVE', 'device_id': '0', " "'gpu_external_alloc': '0', 'enable_cuda_graph': '0', 'gpu_mem_limit': " "'18446744073709551615', 'gpu_external_free': '0', " "'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', " "'cudnn_conv_use_max_workspace': '1', 'cudnn_conv1d_pad_to_nc1d': '0', " "'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', " "'enable_skip_layer_norm_strict_mode': '0'}, 'CPUExecutionProvider': {}}\n" 'find model: ' 'C:\A1111\stable-diffusion-webui\models\faceswaplab\analysers\models\buffalo_l\genderage.onnx ' "genderage ['None', 3, 96, 96] 0.0 1.0\n" "Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with " "options: {'CUDAExecutionProvider': {'do_copy_in_default_stream': '1', " "'cudnn_conv_algo_search': 'EXHAUSTIVE', 'device_id': '0', " "'gpu_external_alloc': '0', 'enable_cuda_graph': '0', 'gpu_mem_limit': " "'18446744073709551615', 'gpu_external_free': '0', " "'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', " "'cudnn_conv_use_max_workspace': '1', 'cudnn_conv1d_pad_to_nc1d': '0', " "'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', " "'enable_skip_layer_norm_strict_mode': '0'}, 'CPUExecutionProvider': {}}\n" 'find model: ' 'C:\A1111\stable-diffusion-webui\models\faceswaplab\analysers\models\buffalo_l\w600k_r50.onnx ' "recognition ['None', 3, 112, 112] 127.5 127.5\n" 'set det-size: (640, 640)\n') C:\A1111\stable-diffusion-webui\venv\lib\site-packages\insightface\utils\transform.py:68: FutureWarning: rcond parameter will change to the default of machine precision times max(M, N) where M and N are the input matrix dimensions. To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1. P = np.linalg.lstsq(X_homo, Y)[0].T # Affine matrix. 3 x 4 2023-08-07 15:02:52,674 - FaceSwapLab - INFO - blend all faces together 2023-08-07 15:02:52,674 - FaceSwapLab - INFO - loading face Shelley.safetensors 2023-08-07 15:02:52,676 - FaceSwapLab - INFO - Process face 0 2023-08-07 15:02:52,681 - FaceSwapLab - INFO - Source Gender 0 Loading swap model: 100%|█████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1.43model/s] 2023-08-07 15:02:53,381 - FaceSwapLab - INFO - ("Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with " "options: {'CUDAExecutionProvider': {'do_copy_in_default_stream': '1', " "'cudnn_conv_algo_search': 'EXHAUSTIVE', 'device_id': '0', " "'gpu_external_alloc': '0', 'enable_cuda_graph': '0', 'gpu_mem_limit': " "'18446744073709551615', 'gpu_external_free': '0', " "'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', " "'cudnn_conv_use_max_workspace': '1', 'cudnn_conv1d_pad_to_nc1d': '0', " "'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', " "'enable_skip_layer_norm_strict_mode': '0'}, 'CPUExecutionProvider': {}}\n" 'inswapper-shape: [1, 3, 128, 128]\n') 2023-08-07 15:02:53,381 - FaceSwapLab - INFO - Target faces count : 1 2023-08-07 15:02:53,381 - FaceSwapLab - INFO - swap face 0 2023-08-07 15:02:56,468 - FaceSwapLab - INFO - **** 2023-08-07 15:02:56,468 - FaceSwapLab - INFO - Inswapper 2023-08-07 15:02:56,469 - FaceSwapLab - WARNING - Please note that improved mask does not work well without upscaling. Set upscaling to Lanczos at least if you want speed and want to use improved mask. 2023-08-07 15:02:56,469 - FaceSwapLab - INFO - improved_mask 2023-08-07 15:02:56,877 - FaceSwapLab - INFO - **** 2023-08-07 15:02:57,052 - FaceSwapLab - INFO - Finished processing image, return 1 images 2023-08-07 15:02:57,053 - FaceSwapLab - INFO - 1 images swapped 2023-08-07 15:02:57,054 - FaceSwapLab - INFO - Upscale with TGHQFace8x_500k scale = 1 2023-08-07 15:03:13,789 - FaceSwapLab - INFO - Restore face with GFPGAN 2023-08-07 15:03:16,034 - FaceSwapLab - INFO - Inpainting face Sampler : DPM++ 2M SDE Karras inpainting_denoising_strength : 0.03 inpainting_steps : 20

2023-08-07 15:03:16,034 - FaceSwapLab - INFO - send faces to image to image 2023-08-07 15:03:16,336 - FaceSwapLab - INFO - Denoising prompt : Portrait of a woman 2023-08-07 15:03:16,337 - FaceSwapLab - INFO - Denoising strenght : 0.03 0%| | 0/1 [00:01<?, ?it/s] 2023-08-07 15:03:18,239 - FaceSwapLab - ERROR - Failed to apply inpainting to face : local variable 'h' referenced before assignment Traceback (most recent call last): File "C:\A1111\stable-diffusion-webui\extensions\sd-webui-faceswaplab\scripts\faceswaplab_inpainting\i2i_pp.py", line 83, in img2img_diffusion i2i_processed = processing.process_images(i2i_p) File "C:\A1111\stable-diffusion-webui\modules\processing.py", line 677, in process_images res = process_images_inner(p) File "C:\A1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack return getattr(processing, 'controlnet_original_process_images_inner')(p, *args, kwargs) File "C:\A1111\stable-diffusion-webui\modules\processing.py", line 794, in process_images_inner samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts) File "C:\A1111\stable-diffusion-webui\modules\processing.py", line 1381, in sample samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning) File "C:\A1111\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 434, in sample_img2img samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) File "C:\A1111\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 303, in launch_sampling return func() File "C:\A1111\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 434, in samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, *extra_params_kwargs)) File "C:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "C:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 650, in sample_dpmpp_2m_sde h_last = h UnboundLocalError: local variable 'h' referenced before assignment 2023-08-07 15:03:18,241 - FaceSwapLab - ERROR - Failed to post-process local variable 'h' referenced before assignment Traceback (most recent call last):** File "C:\A1111\stable-diffusion-webui\extensions\sd-webui-faceswaplab\scripts\faceswaplab_postprocessing\postprocessing.py", line 44, in enhance_image result_image = img2img_diffusion( File "C:\A1111\stable-diffusion-webui\extensions\sd-webui-faceswaplab\scripts\faceswaplab_inpainting\i2i_pp.py", line 97, in img2img_diffusion raise e File "C:\A1111\stable-diffusion-webui\extensions\sd-webui-faceswaplab\scripts\faceswaplab_inpainting\i2i_pp.py", line 83, in img2img_diffusion i2i_processed = processing.process_images(i2i_p) File "C:\A1111\stable-diffusion-webui\modules\processing.py", line 677, in process_images res = process_images_inner(p) File "C:\A1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack return getattr(processing, 'controlnet_original_process_images_inner')(p, *args, kwargs) File "C:\A1111\stable-diffusion-webui\modules\processing.py", line 794, in process_images_inner samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts) File "C:\A1111\stable-diffusion-webui\modules\processing.py", line 1381, in sample samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning) File "C:\A1111\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 434, in sample_img2img samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) File "C:\A1111\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 303, in launch_sampling return func() File "C:\A1111\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 434, in samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, *extra_params_kwargs)) File "C:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "C:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 650, in sample_dpmpp_2m_sde h_last = h UnboundLocalError: local variable 'h' referenced before assignment** 2023-08-07 15:03:18,244 - FaceSwapLab - INFO - Add swp image to processed

Crux1y commented 11 months ago

Before Swap: 00026-3606309543

After: 00027-3606309543-swapped

glucauze commented 11 months ago

Yes, I know that one. It's not technically a bug. It's when improved mask is used with no Upscaler is selected in post-processing (in faces unit). You need to either disable improved mask or add an upscaler (Lanczos if you want speed, LDSR if you want quality, or see configuration below).

See this PR : https://github.com/glucauze/sd-webui-faceswaplab/pull/35 where i describe the same problem. I added a warning in the console to say that it wasn't a good idea to use the improved mask without using an upscaler. You should at least use Lanczos.

 2023-08-07 15:02:56,469 - FaceSwapLab - WARNING - Please note that improved mask does not work well without upscaling. Set upscaling to Lanczos at least if you want speed and want to use improved mask.

I'm talking about the post-processing tuning in the faces. Not global post-processing.

glucauze commented 11 months ago

See here https://github.com/glucauze/sd-webui-faceswaplab/issues/25 recommanded settings.

Crux1y commented 11 months ago

Ah, did the default setting change recently that would enabled the improved mask or deselected a default upscaler?

I fairly certain I've been using the same workflow for the last few days and this configuration problem just started for me.

glucauze commented 11 months ago

Yes it's my fault, i commited too fast. And i was also surprised by the result. The improved mask was set to True by default in the settings (False if no settings were sets). You must have changed the settings in the meantime, which activated the improved mask by default.

I always automatically test the software before committing. But Gradio is a pain to test and there are no automated tests yet. The problem with this kind of little thing is that when I test, my test program uses my configuration. I'd have to do more exhaustive tests, but that would be a real pain.

Crux1y commented 11 months ago

No worries, glad to hear it's just configuration related. Thanks for the quick feedback and info! Feel free to delete/close this thread.

Crux1y commented 11 months ago

I was able to get everything working perfectly again (even better with your suggestions). Thanks again!

Also, I forgot to mention... this is an absolutely amazing extension with a ton of great tools/features. Fantastic job, I really appreciate your hard work!