search

Acly / krita-ai-diffusion

Streamlined interface for generating images with AI in Krita. Inpaint and outpaint with optional text prompt, no tweaking required.
https://www.interstice.cloud
GNU General Public License v3.0
6.59k stars 317 forks source link

CUDA ERROR #24

Closed Mrdanghoang closed 11 months ago

Mrdanghoang commented 1 year ago

Hi guys, I run local, and I got this message when I run 1111 I did install everything dd MY GPU is 4GB, is it acceptable?

Acly commented 1 year ago

This might be an issue with the GPU architecture being too old, but I'm not sure.

Can you attach the server.log file (find it via "View log files" in connection settings)? It has some more information about the error and GPU model.

4GB works to some extent, but it's at the very low end and might cause issues with some workflows.

Mrdanghoang commented 1 year ago

This might be an issue with the GPU architecture being too old, but I'm not sure.

Can you attach the server.log file (find it via "View log files" in connection settings)? It has some more information about the error and GPU model.

4GB works to some extent, but it's at the very low end and might cause issues with some workflows.

2023-10-10 07:56:39,224 INFO Total VRAM 4096 MB, total RAM 8053 MB 2023-10-10 07:56:39,224 INFO Trying to enable lowvram mode because your GPU seems to have 4GB or less. If you don't want this use: --normalvram 2023-10-10 07:56:40,539 INFO Set vram state to: LOW_VRAM 2023-10-10 07:56:40,539 INFO Device: cuda:0 NVIDIA GeForce GTX 950M : cudaMallocAsync 2023-10-10 07:56:40,539 INFO VAE dtype: torch.float32 2023-10-10 07:56:40,600 INFO Using pytorch cross attention 2023-10-10 07:56:42,880 INFO Registered sys.path: ['C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\init.py', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_pycocotools', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_oneformer', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mmpkg', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_midas_repo', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_detectron2', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\python310.zip', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages\win32', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages\win32\lib', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages\Pythonwin'] 2023-10-10 07:56:42,945 INFO 2023-10-10 07:56:42,945 INFO Import times for custom nodes: 2023-10-10 07:56:42,945 INFO 0.0 seconds: C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\custom_nodes\comfyui-tooling-nodes 2023-10-10 07:56:42,945 INFO 0.0 seconds: C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus 2023-10-10 07:56:42,945 INFO 0.3 seconds: C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\custom_nodes\comfyui_controlnet_aux 2023-10-10 07:56:42,946 INFO 2023-10-10 07:56:42,946 INFO 2023-10-10 07:56:42,946 INFO WARNING: this card most likely does not support cuda-malloc, if you get "CUDA error" please run ComfyUI with: --disable-cuda-malloc 2023-10-10 07:56:42,946 INFO 2023-10-10 07:56:42,946 INFO Starting server 2023-10-10 07:56:42,946 INFO 2023-10-10 07:56:42,946 INFO To see the GUI go to: http://127.0.0.1:8188 2023-10-10 07:57:25,913 INFO got prompt 2023-10-10 07:57:26,287 INFO !!! Exception during processing !!! 2023-10-10 07:57:26,287 INFO Traceback (most recent call last): 2023-10-10 07:57:26,287 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\execution.py", line 152, in recursive_execute 2023-10-10 07:57:26,287 INFO output_data, output_ui = get_output_data(obj, input_data_all) 2023-10-10 07:57:26,287 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\execution.py", line 82, in get_output_data 2023-10-10 07:57:26,288 INFO return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) 2023-10-10 07:57:26,288 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\execution.py", line 75, in map_node_over_list 2023-10-10 07:57:26,288 INFO results.append(getattr(obj, func)(slice_dict(input_data_all, i))) 2023-10-10 07:57:26,288 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\nodes.py", line 476, in load_checkpoint 2023-10-10 07:57:26,288 INFO out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings")) 2023-10-10 07:57:26,288 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\comfy\sd.py", line 428, in load_checkpoint_guess_config 2023-10-10 07:57:26,289 INFO model = model_config.get_model(sd, "model.diffusion_model.", device=inital_load_device) 2023-10-10 07:57:26,289 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\comfy\supported_models_base.py", line 48, in get_model 2023-10-10 07:57:26,289 INFO out = model_base.BaseModel(self, model_type=self.model_type(state_dict, prefix), device=device) 2023-10-10 07:57:26,289 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\comfy\model_base.py", line 24, in init 2023-10-10 07:57:26,289 INFO self.diffusion_model = UNetModel(unet_config, device=device) 2023-10-10 07:57:26,289 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 382, in init 2023-10-10 07:57:26,289 INFO operations.Linear(model_channels, time_embed_dim, dtype=self.dtype, device=device), 2023-10-10 07:57:26,289 INFO File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\comfy\ops.py", line 11, in init 2023-10-10 07:57:26,289 INFO self.weight = torch.nn.Parameter(torch.empty((out_features, in_features), **factory_kwargs)) 2023-10-10 07:57:26,290 INFO RuntimeError: CUDA error: operation not supported 2023-10-10 07:57:26,290 INFO CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 2023-10-10 07:57:26,290 INFO For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2023-10-10 07:57:26,290 INFO Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. 2023-10-10 07:57:26,290 INFO 2023-10-10 07:57:26,290 INFO 2023-10-10 07:57:26,290 INFO Prompt executed in 0.33 seconds

Mrdanghoang commented 1 year ago

here is client log

File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\execution.py", line 152, in recursive_execute\n output_data, output_ui = get_output_data(obj, input_data_all)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\execution.py", line 82, in get_output_data\n return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\execution.py", line 75, in map_node_over_list\n results.append(getattr(obj, func)(slice_dict(input_data_all, i)))\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\nodes.py", line 476, in load_checkpoint\n out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings"))\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\sd.py", line 428, in load_checkpoint_guess_config\n model = model_config.get_model(sd, "model.diffusion_model.", device=inital_load_device)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\supported_models_base.py", line 48, in get_model\n out = model_base.BaseModel(self, model_type=self.model_type(state_dict, prefix), device=device)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\model_base.py", line 24, in init\n self.diffusion_model = UNetModel(unet_config, device=device)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 382, in init\n operations.Linear(model_channels, time_embed_dim, dtype=self.dtype, device=device),\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\ops.py", line 11, in init\n self.weight = torch.nn.Parameter(torch.empty((out_features, in_features), **factory_kwargs))\n'] 2023-10-10 07:56:31,267 INFO Extension initialized, Version: 1.3.0, Python: 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] 2023-10-10 07:56:31,455 INFO Found server installation v1.3.0 at C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server 2023-10-10 07:57:25,647 WARNING Style checkpoint not found, using default dreamshaper_8.safetensors 2023-10-10 07:57:25,647 WARNING Style VAE not found, using default VAE from checkpoint 2023-10-10 07:57:25,647 WARNING Style LoRA not found, skipping 2023-10-10 07:57:26,419 ERROR Job 3cac7542-a5ef-4049-a3a2-58c6da9fb909 failed: CUDA error: operation not supported CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\execution.py", line 152, in recursive_execute\n output_data, output_ui = get_output_data(obj, input_data_all)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\execution.py", line 82, in get_output_data\n return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\execution.py", line 75, in map_node_over_list\n results.append(getattr(obj, func)(slice_dict(input_data_all, i)))\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\nodes.py", line 476, in load_checkpoint\n out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings"))\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\sd.py", line 428, in load_checkpoint_guess_config\n model = model_config.get_model(sd, "model.diffusion_model.", device=inital_load_device)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\supported_models_base.py", line 48, in get_model\n out = model_base.BaseModel(self, model_type=self.model_type(state_dict, prefix), device=device)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\model_base.py", line 24, in init\n self.diffusion_model = UNetModel(unet_config, device=device)\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 382, in init\n operations.Linear(model_channels, time_embed_dim, dtype=self.dtype, device=device),\n', ' File "C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy\ops.py", line 11, in init\n self.weight = torch.nn.Parameter(torch.empty((out_features, in_features), **factory_kwargs))\n']

Acly commented 1 year ago

Please try the following: Open settings.json in a text editor (C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\settings.json) Change the line

    "server_arguments": "",

to

    "server_arguments": "--disable-cuda-malloc",
Mrdanghoang commented 1 year ago

Please try the following: Open settings.json in a text editor (C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\settings.json) Change the line

    "server_arguments": "",

to

    "server_arguments": "--disable-cuda-malloc",

sorry but it doesnot work. Same issue

Mrdanghoang commented 1 year ago

but the process takes much longer time, and then the same error

Acly commented 1 year ago

That's somewhat surprising. I noticed this line in server.log:

2023-10-10 07:56:40,539 INFO Device: cuda:0 NVIDIA GeForce GTX 950M : cudaMallocAsync

cudaMallocAsync not working is a known issue with GTX 950M and similar. After you changed the settings.json it should no longer say cudaMallocAsync and usually that fixes the CUDA invalid operation error.

I don't have any other ideas. The GTX 960M in my laptop works okay, but it's a slightly better GPU and still very slow and runs out of VRAM for complex workflows.

Mrdanghoang commented 1 year ago

cudaMallocAsync

2023-10-10 21:34:38,145 INFO Total VRAM 4096 MB, total RAM 8053 MB 2023-10-10 21:34:38,146 INFO Trying to enable lowvram mode because your GPU seems to have 4GB or less. If you don't want this use: --normalvram 2023-10-10 21:34:41,319 INFO Set vram state to: LOW_VRAM 2023-10-10 21:34:41,319 INFO Device: cuda:0 NVIDIA GeForce GTX 950M : native 2023-10-10 21:34:41,319 INFO VAE dtype: torch.float32 2023-10-10 21:34:41,411 INFO Using pytorch cross attention 2023-10-10 21:34:49,117 INFO Registered sys.path: ['C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\init.py', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_pycocotools', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_oneformer', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mmpkg', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_midas_repo', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_detectron2', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\custom_nodes\comfyui_controlnet_aux\src', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI\comfy', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\ComfyUI', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\python310.zip', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages\win32', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages\win32\lib', 'C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion\.server\python\lib\site-packages\Pythonwin'] 2023-10-10 21:34:49,238 INFO 2023-10-10 21:34:49,239 INFO Import times for custom nodes: 2023-10-10 21:34:49,239 INFO 0.0 seconds: C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\custom_nodes\comfyui-tooling-nodes 2023-10-10 21:34:49,239 INFO 0.0 seconds: C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus 2023-10-10 21:34:49,239 INFO 0.7 seconds: C:\Users\Administrator\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI\custom_nodes\comfyui_controlnet_aux 2023-10-10 21:34:49,239 INFO 2023-10-10 21:34:49,240 INFO Starting server 2023-10-10 21:34:49,240 INFO 2023-10-10 21:34:49,240 INFO To see the GUI go to: http://127.0.0.1:8188 2023-10-10 21:35:38,551 INFO got prompt 2023-10-10 21:35:39,332 INFO model_type EPS 2023-10-10 21:35:39,332 INFO adm 0 2023-10-10 21:35:56,585 INFO making attention of type 'vanilla-pytorch' with 512 in_channels 2023-10-10 21:35:56,585 INFO Working with z of shape (1, 4, 32, 32) = 4096 dimensions. 2023-10-10 21:35:56,585 INFO making attention of type 'vanilla-pytorch' with 512 in_channels 2023-10-10 21:36:02,966 INFO missing {'cond_stage_model.logit_scale', 'cond_stage_model.text_projection'} 2023-10-10 21:36:02,968 INFO left over keys: dict_keys(['cond_stage_model.transformer.text_model.embeddings.position_ids']) 2023-10-10 21:36:02,968 INFO loaded straight to GPU 2023-10-10 21:36:02,969 INFO loading new 2023-10-10 21:36:03,020 INFO loading in lowvram mode 256.0 2023-10-10 21:36:03,395 INFO loading new 2023-10-10 21:36:05,524 INFO You shouldn't move a model when it is dispatched on multiple devices.

new logs. The "cudaMallocAsync" disappear, but new error

Mrdanghoang commented 1 year ago

is that anything we can do with it?

2023-10-10 21:54:30,553 INFO For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2023-10-10 21:54:30,553 INFO Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Mrdanghoang commented 1 year ago

do you have discord or something we can chat ??

Acly commented 1 year ago

is that anything we can do with it?

I don't think this will give more useful information, it's deep in 3rd party library code. If there is an incompatibility there, not much I can do about it.

But since it's working on 960M I don't see why 950M shouldn't work too. Did you install the latest GPU drivers? They need to support CUDA 12.1 which is fairly recent. The most basic test is with a 512x512 canvas, no selection, 100% strength. Does that also run into the error?

DigiLiner commented 1 year ago

I have a 950M . I started the server from PowerShell. Then connect to 127.0.0.1:8188 in Krita. Hope this helps.

cd C:\Users\halit\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI ../python/python main.py --disable-cuda-malloc

Mrdanghoang commented 1 year ago

I have a 950M . I started the server from PowerShell. Then connect to 127.0.0.1:8188 in Krita. Hope this helps.

cd C:\Users\halit\AppData\Roaming\krita\pykrita\ai_diffusion.server\ComfyUI ../python/python main.py --disable-cuda-malloc

sorry, Cannot find the python main.py :((

nayzash commented 6 months ago

I have the same problem...