NansException when trying to generate larger images.

[jtran-developer]

Here's the full error message

 0% 0/20 [00:03<?, ?it/s]
Error completing request
Arguments: ('task(681nzcxdv4qmpin)', '', '', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 1800, 1200, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', <scripts.external_code.ControlNetUnit object at 0x7f9625db67f0>, <scripts.external_code.ControlNetUnit object at 0x7f9625db6520>, <scripts.external_code.ControlNetUnit object at 0x7f9625db6880>, <scripts.external_code.ControlNetUnit object at 0x7f9625db60d0>, <scripts.external_code.ControlNetUnit object at 0x7f9625db6910>, <scripts.external_code.ControlNetUnit object at 0x7f9625db61f0>, <scripts.external_code.ControlNetUnit object at 0x7f9625db6550>, <scripts.external_code.ControlNetUnit object at 0x7f9625db6b20>, <scripts.external_code.ControlNetUnit object at 0x7f9625dbb9a0>, <scripts.external_code.ControlNetUnit object at 0x7f9625dbb910>, False, False, 'Horizontal', '1,1', '0.2', False, False, False, 'Attention', False, False, 3, 0, False, False, 0, False, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 20, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, None, False, None, False, None, False, None, False, None, False, None, False, None, False, None, False, None, False, None, False, 50) {}
Traceback (most recent call last):
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/processing.py", line 503, in process_images
    res = process_images_inner(p)
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/processing.py", line 653, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/processing.py", line 869, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 358, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 234, in launch_sampling
    return func()
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 358, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/content/gdrive/MyDrive/sd/stablediffusion/src/k-diffusion/k_diffusion/sampling.py", line 145, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 152, in forward
    devices.test_for_nans(x_out, "unet")
  File "/content/gdrive/.shortcut-targets-by-id/1S33CUH0-cqG4EkR8r0vGFnH9aECswn4r/sd/stable-diffusion-webui/modules/devices.py", line 152, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.

This all started a few days ago. There are no issues when I try to generate smaller images. 1280x1280 is fine, but if I were to go 1200x1800 or 1536x1536, that error is thrown. Basically, I can't hires fix moderately sized images anymore. I'm using the exact same model, loras, settings, prompts and everything else as I always have.

I've tried:

• running it without connecting my gdrive, ensuring it use a freshly downloaded SD with nothing extra
• using the Use_Latest_Working_Commit checkbox
• reverting back to even older versions
• switch to different google accounts Nothing has worked so far.

Maybe something was changed on colab's side? Any help would be appreciated.

[buckwheaton]

I'm getting the same thing. did a clean install, changed model, VAE, lora, and upscaler. same result. and even switched to a premium gpu on colab to see if maybe Google bonked it on their end for the standard gpu. nothing seems to work.

[TheLastBen]

try the runpod notebooks

[TheLastBen]

https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/9294

[buckwheaton]

AUTOMATIC1111/stable-diffusion-webui#9294

In other words, the issue is on the A1111 side and once they fix it the fix will come through to the colab so hold tight? If so, cool, will do. If I'm misunderstanding, let me know. And it's just an annoyance right now rather than a showstopper so I'll just keep rolling with the colab rather than moving to runpod. Thanks again for the notebook!

[TheLastBen]

if A1111 fixes it, it will reflect on the notebook

[jtran-developer]

Thanks, I appreciate you looking into this for us.

[SaturnoElMago]

Same problem since this morning, the colab wont let me generate any images at all, try everything. Ill be checking for updates soon. Thanks for the help @TheLastBen .

[SaturnoElMago]

Same problem since this morning, the colab wont let me generate any images at all, try everything. Ill be checking for updates soon. Thanks for the help @TheLastBen .

[eskaviam]

This notebook is cloning the latest repository but does not have the problem of generating large images. Maybe you guys can use this as a temporary solution until it is fixed.

The downside of this notebook is that the setup can be confusing, and the initial setup can be slow.

[SaturnoElMago]

Thanks @eskaviam! Theres no notebook with ControlNet like this one? Or theres a way to install ControlNet to this notebook? Would be great!

[dkDevHub]

Anyone solved this problem? If there is a solution, please write

[buckwheaton]

Anyone solved this problem? If there is a solution, please write

Still happening for me and I've tried everything I've seen on here or on the A1111 issue thread.

[dkDevHub]

Still happening for me and I've tried everything I've seen on here or on the A1111 issue thread.

so do I.

[buckwheaton]

According to the A1111 issue thread the problem may be with xformers and there is a fix that may be part of the 0.0.19 release.

[buckwheaton]

Still happening for me and I've tried everything I've seen on here or on the A1111 issue thread.

so do I.

Ok the following fixed this, at least for my version of this problem: remove --xformers as a commandline argument and add --opt-sdp-attention. I did this at the bottom of the Start Stable Diffusion section of the colab notebook - show code then scroll all the way down. I made this change in each of the three versions of the command line although I don't really know what I'm doing and probably didn't have to. :). Downside is that xformers is more memory efficient so I can't upscale quite as large in hires fix now but at least it's sort of working.

[jtran-developer]

edit: taking this back, please disregard

[TheLastBen]

@buckwheaton Your fix has been implemented

[Super-zapper]

I tried to add --lowvram and managed to generate 1536x1536 with free GPU without out of memory. Hopefully after xformers fix we could use it and save more vram.

Also I wanted to try to hire better GPU with more vram and generate biggest image possible and I have a question: is it possible to generate images bigger than 2048x2048?

[TheLastBen]

you can make bigger than 2048x2048, but the coherence might not be there, use my runpod template for cheaper GPUs

[Super-zapper]

I want to generate maps, so I hope image will just be extended

I mean how can I set width and height more than this limit here? Also can I ask where can I find your template?

[eskaviam]

I want to generate maps, so I hope image will just be extended

I mean how can I set width and height more than this limit here? Also can I ask where can I find your template?

outpainting would be your best choice

[TheLastBen]

@Super-zapper https://www.runpod.io/console/gpu-secure-cloud?template=runpod-stable-unified

You can also click on the Runpod thumbnail on the main page of this repo, the rtx 3090 is only 0.3$/hour, better than the A100

[Super-zapper]

I want to generate maps, so I hope image will just be extended I mean how can I set width and height more than this limit here? Also can I ask where can I find your template?

outpainting would be your best choice

Thanks, but I hoped to find less limited way

[Super-zapper]

@Super-zapper https://www.runpod.io/console/gpu-secure-cloud?template=runpod-stable-unified

You can also click on the Runpod thumbnail on the main page of this repo, the rtx 3090 is only 0.3$/hour, better than the A100

Thank you

[buckwheaton]

Hey it looks like the new xformers is out. It may resolve the nans errors and could be more memory efficient than --opt-sdp-attention? I'm out of my depth as far as trying it on mine but just wanted to share the news in case anyone here wanted to give it a shot.

[TheLastBen]

I'll check it out EDIT: still not fixed

[toshioson]

I am also getting this when using poor man's outpainting with SD XL checkpoints, adding the argument --no-half makes it working for me but it takes much higher s/it even to just expand 128 pixel on one side.

If I use the --disable-nan-check argument I got black output only...

[TheLastBen]

replace --xformers with --opt-sdp-attention