search

Bing-su / adetailer

Auto detecting, masking and inpainting with detection model.
GNU Affero General Public License v3.0
4.04k stars 316 forks source link

[Bug]: modules.devices.NansException: A tensor with all NaNs was produced in Unet. #140

Closed DLai126 closed 1 year ago

DLai126 commented 1 year ago

Describe the bug

When I use "adetailer" in txt2img to generate an image, I move the image to img2img and use "Tiled Diffusion" to enlarge the image and enhance details using the tile functionality of controlnet. I also use "adetailer" again. However, the following error occurs.

Full console logs

Error completing request                                                                | 1/10 [00:09<01:25,  9.45s/it]
Arguments: ('task(ftizvkaq689lmvr)', 0, "(best quality) (masterpiece), (highly detailed), original, extremely detailed artwork, solo, {intense and enigmatic eyes}, 1 female rogue, immersed in the dark and ominous atmosphere of the Diablo universe. She navigates through a blood-red desert, the sky tinged with an eerie glow. Dressed in stealthy attire and wielding lethal weapons, she exudes a sense of danger and resilience. The artwork captures the treacherous beauty of the crimson desert, with swirling sandstorms, jagged rock formations, and hints of demonic presence. The woman's stance displays a combination of agility and determination, a formidable force amidst the unforgiving landscape", '(worst quality:2), (low quality:2), (normal quality:2), ((monochrome)), ((grayscale)),paintings, sketches, skin spots, acnes, skin blemishes, bad anatomy, facing away, looking away, tilted head, multiple girls, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, blurry, bad feet, cropped, poorly drawn hands, poorly drawn face, mutation, deformed, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, extra fingers, fewer digits, extra limbs, extra arms, extra legs, malformed limbs, fused fingers, too many fingers, long neck, cross-eyed, mutated hands, polar lowres, bad body, bad proportions, gross, extra breasts, ng_deepnegative_v1_75t, bad-hands-5, ', [], <PIL.Image.Image image mode=RGBA size=1920x1072 at 0x21D8ACAEDA0>, None, None, None, None, None, None, 30, 16, 4, 0, 1, False, False, 1, 1, 7, 1.5, 0.3, -1.0, -1.0, 0, 0, 0, False, 0, 1072, 1920, 1, 0, 0, 32, 0, '', '', '', [], 0, True, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1}, True, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'R-ESRGAN 4x+', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, True, 1536, 96, True, True, True, False, False, '', 0, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 7, 100, 'Constant', 0, 'Constant', 0, 4, False, False, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, None, 'Refresh models', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000021D8ABD1540>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000021D8ABD3C40>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000021D8B8D9FF0>, False, 1, 0.15, False, 'OUT', ['OUT'], 5, 0, 'Bilinear', False, 'Bilinear', False, 'Lerp', '', '', False, False, None, True, False, 1, 0.15, False, 'OUT', ['OUT'], 5, 0, 'Bilinear', False, 'Bilinear', False, 'Lerp', '', '', False, '', True, 0, 'values', '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1', 'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False, 'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:0\n\nXYZ:::1', False, False, False, False, '<ul>\n<li><code>CFG Scale</code> should be 2 or lower.</li>\n</ul>\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, None, False, None, None, False, None, None, False, 50, '', 2, True, 'Wavelet', False, '<p style="margin-bottom:0.75em">Will upscale the image depending on the selected target size type</p>', 512, 0, 8, 32, 64, 0.35, 32, 0, True, 0, False, 8, 0, 0, 2048, 2048, 2) {}
Traceback (most recent call last):
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\img2img.py", line 178, in img2img
    processed = process_images(p)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\processing.py", line 610, in process_images
    res = process_images_inner(p)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\processing.py", line 728, in process_images_inner
    samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 350, in process_sample
    return process.sample_before_CN_hack(*args, **kwargs)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\processing.py", line 1261, in sample
    samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 356, in sample_img2img
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 257, in launch_sampling
    return func()
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 356, in <lambda>
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "D:\AI_Factory\stable2\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 169, in forward
    devices.test_for_nans(x_out, "unet")
  File "D:\AI_Factory\stable2\stable-diffusion-webui\modules\devices.py", line 156, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.

List of installed extensions

No response

Bing-su commented 1 year ago

After seeing several cases, I don't believe the NAN issue has anything to do with adetailer, even if the issue only appears when using adetailer.

What we can do is try to change the prompts, models, VAEs, loras, embeddings, and weights we use and see if we can find a point where we don't get that error.

DLai126 commented 1 year ago

After seeing several cases, I don't believe the NAN issue has anything to do with adetailer, even if the issue only appears when using adetailer.

What we can do is try to change the prompts, models, VAEs, loras, embeddings, and weights we use and see if we can find a point where we don't get that error.

Thank you for your response. I have found that sometimes, when keeping the prompts, models, VAEs, loras, and weights the same, enlarging the image will succeed, while other times it will fail. However, I still really like the effects of adetailer, so it seems that I'll have to make do with using it.

freecoderwaifu commented 1 year ago

Could be another issue with the recent Nvidia driver memory changes, but unsure. Only encountered this issue recently too, it was working fine before the driver update. Loading another model fixes it, but only for the first generation of that model, maybe related to how all the different models (yolo, etc., controlnet models, VAE and checkpoint) are loaded into memory.

Arron17 commented 1 year ago

I'm getting similar issues to this, I've noticed that it happens when using the Reference ControlNet to generate the image. If I turn off the controlnet it works as expected.

heroofthebeach commented 1 year ago

EDIT: Adding "--no-half" to the command line did fix things for me and I should've done it ages ago. Only use "--disable-nan-check" as a stopgap if nothing else works. Otherwise, ignore that advice.

The rest of the original message is still relevant though. I don't think this is an ADetailer issue. ADetailer only draws our attention to it because using ADetailer creates a second exposure to whatever process has the bug in it.

I've been encountering this too and my hunch is that it's some kind of memory issue, as it happens more often at larger resolutions. I have the controlnet extension installed but I'm not using it and it's happening to me frequently.

As the error says, it can be overridden by adding "--disable-nan-check" to your webui-user.bat file, but this only changes what happens when the error hits, as the zone failing to be generated will be replaced by a black square. If the error happens during primary generation, the whole image is replaced by a black square, and if it happens while ADetailer is processing, only the face becomes a black square.

It's not a solution at all, but it does mean that when generating multiple images, the process won't completely stop and you can still generate the rest of the images (or at least attempt to). I think this is still an improvement, since I'd rather have eight failed images in a set of 16 than the process canceling itself three images in.

And given that the generation can fail differently at two different stages, I agree that this is not almost certainly not an ADetailer bug and there's probably nothing the ADetailer team can do to address it.

Elise96nl commented 2 months ago

redownload models. put in in seperate folder (fixed models) and give it a hard read only attribute. Use seperate checkpoint for adetailer, so it has to reload between txt2image and adetailer. also allow only 1 model in memory.

I've had no nans errors after doing that