search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
16.95k stars 1.95k forks source link

[Bug]: CUDA error: misaligned address，Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. #1043

Closed tanglangxia closed 1 year ago

tanglangxia commented 1 year ago

Is there an existing issue for this?

What happened?

The latest 22BCC7BE version of the WebUI version is normal to use the picture normally. As long as it runs the controlnet, it will report intermittently. Sometimes it can be used normally, but a report information will appear again several times. The error message is as follows: xception in thread MemMon:

Traceback (most recent call last):
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 53, in run
    free, total = self.cuda_mem_get_info()
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 34, in cuda_mem_get_info
    return torch.cuda.mem_get_info(index)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\cuda\memory.py", line 618, in mem_get_info
    return torch.cuda.cudart().cudaMemGetInfo(device)
RuntimeError: CUDA error: misaligned address
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Try to switch the webui version and the ControlNet version, there will be errors. Try to close other plug -in, and only retain the ControlNet version.

Steps to reproduce the problem

  1. Run webui, fill in keywords, and have tried no keywords
  2. Use the Lineart function of ControlNet
  3. Run the program, the error position is random, sometimes 0%, sometimes 10%, sometimes 50%error, but the content of the error report is consistent

What should have happened?

It should be run normally, rather than collapsed, and cannot be out of the picture again. The premium card is dead. You can only restart the program before you can continue to use

Commit where the problem happens

webui: 22bcc7be controlnet: c5fbfc31（ v1.1.111） python: 3.10.6 torch: 2.0.0+cu118 xformers: N/A gradio: 3.23.0 Loading model from cache: control_v11p_sd15_lineart [43d4be0d]

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

No

Console logs

Loading model from cache: control_v11p_sd15_lineart [43d4be0d]█████████████████████████| 35/35 [00:23<00:00,  1.50it/s]
Loading preprocessor: lineart
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 1024
raw_W = 1024
target_H = 1024
target_W = 1024
estimation = 1024.0
preprocessor resolution = 1024
ControlNet preprocessor location: E:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
  0%|                                                                                           | 0/35 [00:00<?, ?it/s]Exception in thread MemMon:
Traceback (most recent call last):
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 53, in run
    free, total = self.cuda_mem_get_info()
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 34, in cuda_mem_get_info
    return torch.cuda.mem_get_info(index)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\cuda\memory.py", line 618, in mem_get_info
    return torch.cuda.cudart().cudaMemGetInfo(device)
RuntimeError: CUDA error: misaligned address
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Error completing request
Arguments: ('task(59l8f5e75y3yea6)', '((masterpiece, best quality)),Professional car design drawings, (concept art), digital artwork, trending on ArtStation, trending on CGSociety, Intricate, High Detail, Sharp focus, vivid, hyperdetailed, highres, 8k, ultra-detailed, absurdres, (high detail:1.1), hdr, depth of field, strong bloom, anime cel shading, octanerender, sudden ice age, flash freeze, (vibrant colors), vector art, Very long range, wide shot of an extremely detailed, bright and colorful, heavily stylized naturepunk poster, (neon glow), (rich color palette:1.4), high contrast, (best quality:1.4), starry sky, 8k UHD, hd, hi res, cgsociety contest winner, detailed matte painting, psychedelic, comics style, analog style, illustration, lineart, oil painting \\(medium\\), splash art, flat colors, strong bloom, mesmerizing, captivating, (shadows), dynamic perspective, professional, majestic, dramatic, Caustics, PBR, Physically based Rendering, Ray-tracing, volume-marching, Global Illumination, Subsurface Scattering, Iridescence:1.5, epic, cinematic lighting, glowing, volumetric lighting, detailed background, ecopunk, pink full moon, pink pastel style, infrared photography, radioactive environment, luminescent algae, torii, (god rays, dramatic light, intricate), (soft light, rim light:1.2), symmetric composition, (fractal structures, thousands of ribbons, spirals:1.2, dreamlike, muted colors, geometric pattern,', '(worst quality:2), (low quality:2), (normal quality:2), lowres, bad anatomy, bad hands, normal quality, ((monochrome)), ((grayscale))', [], 35, 15, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 1024, 1024, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, <controlnet.py.UiControlNetUnit object at 0x000001C30E052590>, <controlnet.py.UiControlNetUnit object at 0x000001C30A0799C0>, <controlnet.py.UiControlNetUnit object at 0x000001C416CDB7C0>, <controlnet.py.UiControlNetUnit object at 0x000001C416CABF10>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, None, False, None, False, None, False, None, False, 50) {}
Traceback (most recent call last):
  File "E:\AI\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "E:\AI\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\modules\txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "E:\AI\stable-diffusion-webui\modules\processing.py", line 503, in process_images
    res = process_images_inner(p)
  File "E:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "E:\AI\stable-diffusion-webui\modules\processing.py", line 653, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "E:\AI\stable-diffusion-webui\modules\processing.py", line 869, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "E:\AI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "E:\AI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in launch_sampling
    return func()
  File "E:\AI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 147, in forward
    x_out[-uncond.shape[0]:] = self.inner_model(x_in[-uncond.shape[0]:], sigma_in[-uncond.shape[0]:], cond=make_condition_dict([uncond], image_cond_in[-uncond.shape[0]:]))
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "E:\AI\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 277, in forward2
    return forward(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 208, in forward
    control = param.control_model(x=x_in, hint=param.used_hint_cond, timesteps=timesteps, context=context)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 115, in forward
    return self.control_model(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 383, in forward
    h = module(h, emb, context)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
    x = layer(x, context)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
    x = block(x, context=context[i])
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
    return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
    return CheckpointFunction.apply(func, len(inputs), *args)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
    output_tensors = ctx.run_function(*ctx.input_tensors)
  File "E:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 272, in _forward
    x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\AI\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 139, in split_cross_attention_forward
    r2 = rearrange(r1, '(b h) n d -> b n (h d)', h=h)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 487, in rearrange
    return reduce(tensor, pattern, reduction='rearrange', **axes_lengths)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 410, in reduce
    return _apply_recipe(recipe, tensor, reduction_type=reduction)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 239, in _apply_recipe
    return backend.reshape(tensor, final_shapes)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\_backends.py", line 84, in reshape
    return x.reshape(shape)
RuntimeError: CUDA error: misaligned address
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
    output = await app.get_blocks().process_api(
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1075, in process_api
    result = await self.call_function(
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 884, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "E:\AI\stable-diffusion-webui\modules\call_queue.py", line 92, in f
    mem_stats = {k: -(v//-(1024*1024)) for k, v in shared.mem_mon.stop().items()}
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 92, in stop
    return self.read()
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 77, in read
    free, total = self.cuda_mem_get_info()
  File "E:\AI\stable-diffusion-webui\modules\memmon.py", line 34, in cuda_mem_get_info
    return torch.cuda.mem_get_info(index)
  File "E:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\cuda\memory.py", line 618, in mem_get_info
    return torch.cuda.cudart().cudaMemGetInfo(device)
RuntimeError: CUDA error: misaligned address
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Additional information

At present, you try to switch all versions of WebUI and ControlNet, uninstall the environment, and re -configure the environment. This problem has occurred, so it is not clear whether it is a software level. By the way, list the list of PIP

Package Version

absl-py 1.4.0 accelerate 0.12.0 addict 2.4.0 aenum 3.1.12 aiofiles 23.1.0 aiohttp 3.8.4 aiosignal 1.3.1 altair 4.2.2 antlr4-python3-runtime 4.9.3 anyio 3.6.2 async-timeout 4.0.2 attrs 23.1.0 av 10.0.0 basicsr 1.4.2 beautifulsoup4 4.12.2 blendmodes 2022 boltons 23.0.0 cachetools 5.3.0 certifi 2022.12.7 cffi 1.15.1 chardet 4.0.0 charset-normalizer 3.1.0 clean-fid 0.1.29 click 8.1.3 clip 1.0 colorama 0.4.6 contourpy 1.0.7 cssselect2 0.7.0 cycler 0.11.0 decorator 4.0.11 deprecation 2.1.0 duckduckgo-search 2.8.0 einops 0.4.1 entrypoints 0.4 facexlib 0.3.0 fastapi 0.94.0 ffmpy 0.3.0 filelock 3.12.0 filterpy 1.4.5 flatbuffers 23.3.3 font-roboto 0.0.1 fonts 0.0.3 fonttools 4.39.3 frozenlist 1.3.3 fsspec 2023.4.0 ftfy 6.1.1 future 0.18.3 fvcore 0.1.5.post20221221 gdown 4.7.1 gfpgan 1.3.8 gitdb 4.0.10 GitPython 3.1.30 google-auth 2.17.3 google-auth-oauthlib 1.0.0 gradio 3.23.0 grpcio 1.54.0 h11 0.12.0 httpcore 0.15.0 httpx 0.24.0 huggingface-hub 0.14.0 idna 2.10 imageio 2.28.0 imageio-ffmpeg 0.4.8 inflection 0.5.1 iopath 0.1.9 Jinja2 3.1.2 jsonmerge 1.8.0 jsonschema 4.17.3 kiwisolver 1.4.4 kornia 0.6.7 lark 1.1.2 lazy_loader 0.2 lightning-utilities 0.8.0 linkify-it-py 2.0.0 llvmlite 0.39.1 lmdb 1.4.1 lpips 0.1.4 lxml 4.9.2 Markdown 3.4.3 markdown-it-py 2.2.0 MarkupSafe 2.1.2 matplotlib 3.7.1 mdit-py-plugins 0.3.3 mdurl 0.1.2 mediapipe 0.9.3.0 moviepy 0.2.3.2 mpmath 1.3.0 multidict 6.0.4 networkx 3.1 numba 0.56.4 numexpr 2.8.4 numpy 1.23.3 oauthlib 3.2.2 omegaconf 2.2.3 open-clip-torch 2.7.0 opencv-contrib-python 4.7.0.72 opencv-python 4.7.0.72 orjson 3.8.10 packaging 23.1 pandas 2.0.1 piexif 1.1.3 Pillow 9.4.0 pip 22.2.1 portalocker 2.7.0 protobuf 3.20.0 psutil 5.9.5 pyasn1 0.5.0 pyasn1-modules 0.3.0 pycparser 2.21 pycryptodome 3.17 pydantic 1.10.7 pyDeprecate 0.3.2 pydub 0.25.1 pyparsing 3.0.9 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-multipart 0.0.6 pytorch-lightning 1.9.4 pytz 2023.3 PyWavelets 1.4.1 pywin32 306 PyYAML 6.0 realesrgan 0.3.0 regex 2023.3.23 reportlab 3.6.12 requests 2.25.1 requests-oauthlib 1.3.1 resize-right 0.0.2 rsa 4.9 safetensors 0.3.0 scikit-image 0.19.2 scipy 1.10.1 semantic-version 2.10.0 sentencepiece 0.1.98 setuptools 63.2.0 six 1.16.0 smmap 5.0.0 sniffio 1.3.0 sounddevice 0.4.6 soupsieve 2.4.1 starlette 0.26.1 svglib 1.5.1 sympy 1.11.1 tabulate 0.9.0 tb-nightly 2.13.0a20230424 tensorboard 2.12.2 tensorboard-data-server 0.7.0 tensorboard-plugin-wit 1.8.1 termcolor 2.3.0 tifffile 2023.4.12 timm 0.6.7 tinycss2 1.2.1 tokenizers 0.13.3 tomli 2.0.1 toolz 0.12.0 torch 2.0.0+cu118 torchdiffeq 0.2.3 torchmetrics 0.11.4 torchsde 0.2.5 torchvision 0.15.1+cu118 tqdm 4.65.0 trampoline 0.1.2 transformers 4.25.1 typing_extensions 4.5.0 tzdata 2023.3 uc-micro-py 1.0.1 urllib3 1.26.15 uvicorn 0.21.1 wcwidth 0.2.6 webencodings 0.5.1 websockets 11.0.2 Werkzeug 2.2.3 wheel 0.40.0 xformers 0.0.17 yacs 0.1.8 yapf 0.33.0 yarl 1.9.1

lllyasviel commented 1 year ago

use 512x512 as resolution. your device capability is not high enough for 1024

tanglangxia commented 1 year ago

use 512x512 as resolution. your device capability is not high enough for 1024

I used the picture of 3090. The 24G was not completely used. I only used a half of the capacity.

lllyasviel commented 1 year ago

from log, this problem is not related to controlnet. please check if your a1111 env has problems.

tanglangxia commented 1 year ago

from log, this problem is not related to controlnet. please check if your a1111 env has problems. Thank you, I have adjusted the ENV of the webui, although it has been deleted and re -configured many times,

Wite905 commented 1 year ago

did you find a solution to the problem?

Wite905 commented 1 year ago

i have the same problem, when i use the deforum :( i have 1080ti also got this error in 256*256 i didnt use --medvram sometimes it works sometimes not

Only windows reinstalling helped me

tanglangxia commented 1 year ago

Unfortunately, I re -installed Windows and did not solve this problem. I have doubted whether there is a problem with the GPU