[Bug]: Upscaling: CUDA illegal memory access

[jaggzh]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Each time I try Extras -> Upscale -> Generate I get this crash, even from a fresh load of the webui.

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

(There is this illegal access issue but that one isn't seemingly related to upscaling: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/2955)

Steps to reproduce the problem

1. Pick an image from the Image Browser extension, Send to Extras
2. Select ESRGAN_4x and whatever other options you want (it doesn't seem to matter which I choose).
3. Click Generate

What should have happened?

It should have blessed me with miraculous upscaling love.

Commit where the problem happens

a8a12df56986121a4fcdf59c3fc21fd10fa3ad6a

What platforms do you use to access UI ?

Linux

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--allow-code --medvram --enable-console-prompts --listen

Additional information, context and logs

Error completing request
Arguments: (0, 0, <PIL.Image.Image image mode=RGB size=704x384 at 0x7F50E831F280>, None, '', '', True, 0, 0, 0, 2, 512, 512, True, 3, 0, 1, False) {}
Traceback (most recent call last):
  File "/home/jaggz/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/home/jaggz/stable-diffusion-webui/webui.py", line 51, in f
    shared.state.begin()
  File "/home/jaggz/stable-diffusion-webui/modules/shared.py", line 187, in begin
    devices.torch_gc()
  File "/home/jaggz/stable-diffusion-webui/modules/devices.py", line 36, in torch_gc
    torch.cuda.empty_cache()
  File "/home/jaggz/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/cuda/memory.py", line 121, in empty_cache
    torch._C._cuda_emptyCache()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[JohnWJarrett]

I get this error just randomly when making txt2img prompts, sometimes it can go hundreds of gens and not happen and sometimes it just happens on the first gen. So not just up-scaling. Exact same error

[Thakshara9728]

same error I can't even generate any image now 😢 My GPU Rtx 3060TI

[neojam]

@Thakshara9728 Did you perhaps install dreambooth extention? I always got this error when promting after installing it. Had to remove the extension to make the SD work again. The extensions replaces 0.3.0 diffusers with 0.8.0 version under "\venv\Lib\site-packages\diffusers", which breaks webui for me...

[Leslie-d-a]

I got the same error after generating around 4 images with txt2img. I'm running a 1050ti 4g with args: --precision full --no-half --lowvram --opt-split-attention. I've already done a clean install but it happens again after a short while

[0xItx]

Are you undervolting and/or overclocking your GPU? In my experience if your clock/voltage settings are not 100% stable you sometimes get random CUDA errors like these.

[Leslie-d-a]

@0xItx yep I did have a small overclock on my gpu, been running it for a while now and no issues popped up. thanks a lot!

[Thakshara9728]

@Thakshara9728 Did you perhaps install dreambooth extention? I always got this error when promting after installing it. Had to remove the extension to make the SD work again. The extensions replaces 0.3.0 diffusers with 0.8.0 version under "\venv\Lib\site-packages\diffusers", which breaks webui for me...

No I didn't installed any extensions .just clean SD 1.5

[DeXtmL]

I think this might be related to unclean finish of the previous generation. Some old cuda threads and buffers are still active when you hit the next Generate.

[Sniper199999]

Are you undervolting and/or overclocking your GPU? In my experience if your clock/voltage settings are not 100% stable you sometimes get random CUDA errors like these.

This might be what is happening to my 3060. I have an undervolt of 850mV by default and I started getting CUDA errors. I need to test Automatic1111 on default settings again to confirm whether its an under-volt issue.

[TheOnlyHolyMoly]

Please advise if issue is solved now.

[lhw11]

I also encountered the same problem, have you resolved it

[Sniper199999]

the issue is probably hardware related, maybe GPU is faulty, or you have applied some overclocking/under-volting to your GPU. I actually solved this issue by reversing by under-volt to the normal profile of my GPU.

[hkpD]

Check MSI Afterburner, too. I got mine to work after switching profiles.