"CUDA_PATH is set but CUDA wasn't able to be loaded"

michael-pptf commented 9 months ago

First, confirm

[X] I have read the instruction carefully
[X] I have searched the existing issues
[X] I have updated the extension to the latest version

What happened?

Runtime error when ComfyUI reaches Reactor.

Steps to reproduce the problem

Your workflow

Using ComfyUI Portable, release 11-28-2023
Since this version uses Python 3.11, manual installation steps were followed to install pre-built wheel.
Import Reactor Node after image generation. Reference image was provided, face_model is empty since I don't have one.
When the progress reaches reactor, the error errored.

Sysinfo

Windows 10 LTSC, AMD Ryzen 7 7700X + RTX 3090 ImpactPack Controlnet Davemane42 Derfuu_Nodes facerestore_cf rgthree

Relevant console log

Starting server

To see the GUI go to: http://0.0.0.0:8188
FETCH DATA from: C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json
FETCH DATA from: C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json
got prompt
model_type EPS
adm 2816
Using xformers attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using xformers attention in VAE
missing {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Requested to load SDXLClipModel
Loading 1 new model
Requested to load SDXL
Loading 1 new model
100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:13<00:00,  1.88it/s]
Requested to load AutoencoderKL
Loading 1 new model
[ReActor] 18:53:05 - STATUS - Working: source face index [0], target face index [0]
[ReActor] 18:53:05 - STATUS - Analyzing Source Image...
2023-11-30 18:53:05.4221425 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

EP Error D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.
 when using ['CUDAExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
2023-11-30 18:53:05.4816392 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 463, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\nodes.py", line 117, in execute
    script.process(
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_faceswap.py", line 96, in process
    result = swap_face(
             ^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 195, in swap_face
    source_faces = analyze_faces(source_img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 114, in analyze_faces
    face_analyser = copy.deepcopy(getAnalysisModel())
                                  ^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 69, in getAnalysisModel
    ANALYSIS_MODEL = insightface.app.FaceAnalysis(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\reactor_log_patch.py", line 48, in patched_faceanalysis_init
    model = model_zoo.get_model(onnx_file, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\insightface\model_zoo\model_zoo.py", line 96, in get_model
    model = router.get_model(providers=providers, provider_options=provider_options)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\reactor_log_patch.py", line 21, in patched_get_model
    session = PickableInferenceSession(self.onnx_file, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\insightface\model_zoo\model_zoo.py", line 25, in __init__
    super().__init__(model_path, **kwargs)
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 430, in __init__
    raise fallback_error from e
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 425, in __init__
    self._create_inference_session(self._fallback_providers, None)
  File "C:\Users\michael\Documents\comfyui\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 463, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

Prompt executed in 17.21 seconds
gc collect

Additional information

No response

Gourieff commented 9 months ago

I think there can be two probable reasons:

Smth's wrong with inswapper_128 model file (try to download it again)
or you can try this solution: https://github.com/Gourieff/sd-webui-reactor/issues/200#issuecomment-1813805022 python_embeded\python.exe pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

ilcristofer commented 9 months ago

Hello get this error:

[ReActor] 01:39:13 - STATUS - Working: source face index [0], target face index [0] [ReActor] 01:39:14 - STATUS - Analyzing Source Image... 2023-12-04 01:39:14.4227271 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

EP Error D:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported. when using ['CUDAExecutionProvider'] Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying. 2023-12-04 01:39:14.5799241 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

ERROR:root:!!! Exception during processing !!! ERROR:root:Traceback (most recent call last): File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 463, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) RuntimeError: D:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\nodes.py", line 117, in execute script.process( File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_faceswap.py", line 96, in process result = swap_face( ^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 195, in swap_face source_faces = analyze_faces(source_img) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 114, in analyze_faces face_analyser = copy.deepcopy(getAnalysisModel()) ^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 69, in getAnalysisModel ANALYSIS_MODEL = insightface.app.FaceAnalysis( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\reactor_log_patch.py", line 48, in patched_faceanalysis_init model = model_zoo.get_model(onnx_file, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\insightface\model_zoo\model_zoo.py", line 96, in get_model model = router.get_model(providers=providers, provider_options=provider_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node\reactor_log_patch.py", line 21, in patched_get_model session = PickableInferenceSession(self.onnx_file, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\insightface\model_zoo\model_zoo.py", line 25, in init super().init(model_path, kwargs) File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 430, in init raise fallback_error from e File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 425, in init self._create_inference_session(self._fallback_providers, None) File "C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 463, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) RuntimeError: D:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

Tried python_embeded\python.exe pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

get this: PS C:\Users\Roberto\stable\ComfyUI_windows_portable> python_embeded\python.exe pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 C:\Users\Roberto\stable\ComfyUI_windows_portable\python_embeded\python.exe: can't open file 'C:\Users\Roberto\stable\ComfyUI_windows_portable\pip': [Errno 2] No such file or directory PS C:\Users\Roberto\stable\ComfyUI_windows_portable>

Gourieff commented 9 months ago

Hello! Sorry, my fault... Missed "-m" flag before pip 🤦‍♂️

python_embeded\python.exe -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

4DeepThought2 commented 9 months ago

Im actually have the exact same issue right now:

Error occurred when executing ReActorFaceSwap:

D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:743 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

File "Y:\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "Y:\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "Y:\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "Y:\ComfyUI\custom_nodes\comfyui-reactor-node\nodes.py", line 117, in execute
script.process(
File "Y:\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_faceswap.py", line 96, in process
result = swap_face(
File "Y:\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 195, in swap_face
source_faces = analyze_faces(source_img)
File "Y:\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 114, in analyze_faces
face_analyser = copy.deepcopy(getAnalysisModel())
File "Y:\ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py", line 69, in getAnalysisModel
ANALYSIS_MODEL = insightface.app.FaceAnalysis(
File "Y:\ComfyUI\venv\lib\site-packages\insightface\app\face_analysis.py", line 31, in __init__
model = model_zoo.get_model(onnx_file, **kwargs)
File "Y:\ComfyUI\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 96, in get_model
model = router.get_model(providers=providers, provider_options=provider_options)
File "Y:\ComfyUI\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 40, in get_model
session = PickableInferenceSession(self.onnx_file, **kwargs)
File "Y:\ComfyUI\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 25, in __init__
super().__init__(model_path, **kwargs)
File "Y:\ComfyUI\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 430, in __init__
raise fallback_error from e
File "Y:\ComfyUI\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 425, in __init__
self._create_inference_session(self._fallback_providers, None)
File "Y:\ComfyUI\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 463, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)

Running in a venv of course. Here is what the venv sees:

(venv) PS Y:\ComfyUI> nvidia-smi
Mon Dec  4 14:11:25 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.01                 Driver Version: 546.01       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090      WDDM  | 00000000:01:00.0  On |                  N/A |
| 59%   67C    P2             303W / 420W |   9687MiB / 24576MiB |     52%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

Not sure why its not loading cuda

ilcristofer commented 9 months ago

Hello! Sorry, my fault... Missed "-m" flag before pip 🤦‍♂️

python_embeded\python.exe -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

Hello. I tried but this is what it tells me: PS C:\Users\Roberto\stable\ComfyUI_01> python_embeded\python.exe -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 Looking in indexes: https://download.pytorch.org/whl/cu118 Requirement already satisfied: torch in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (2.1.1+cu121) Requirement already satisfied: torchvision in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (0.16.1+cu121) Requirement already satisfied: filelock in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torch) (3.13.1) Requirement already satisfied: typing-extensions in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torch) (4.8.0) Requirement already satisfied: sympy in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torch) (1.12) Requirement already satisfied: networkx in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torch) (3.2.1) Requirement already satisfied: jinja2 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torch) (3.1.2) Requirement already satisfied: fsspec in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torch) (2023.10.0) Requirement already satisfied: numpy in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torchvision) (1.26.2) Requirement already satisfied: requests in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torchvision) (2.31.0) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from torchvision) (9.5.0) Requirement already satisfied: MarkupSafe>=2.0 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from jinja2->torch) (2.1.3) Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from requests->torchvision) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from requests->torchvision) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from requests->torchvision) (1.26.18) Requirement already satisfied: certifi>=2017.4.17 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from requests->torchvision) (2023.11.17) Requirement already satisfied: mpmath>=0.19 in c:\users\roberto\stable\comfyui_01\python_embeded\lib\site-packages (from sympy->torch) (1.3.0) PS C:\Users\Roberto\stable\ComfyUI_01>

4DeepThought2 commented 9 months ago

Not sure why its not loading cuda

The environment doesnt seem to know about it:

(venv) PS Y:\ComfyUI> python
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2023-12-04 15:01:59.300300: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING:tensorflow:From Y:\ComfyUI\venv\lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

>>> tf.config.list_physical_devices('GPU')
[]

Gourieff commented 9 months ago

CUDA Execution Provider requires CUDA 11.8 https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html You guys just need to download the appropriate build of Comfy This one for example (latest with CU11.8): https://github.com/comfyanonymous/ComfyUI/releases/download/latest/ComfyUI_windows_portable_nvidia_cu118_or_cpu_23_09_2023.7z Or any other you want but with cu118 https://github.com/comfyanonymous/ComfyUI/releases

Gourieff commented 9 months ago

I will add it to the documentation

4DeepThought2 commented 9 months ago

Ah, I gotcha, the current iteration of reactor-node requests cuda11? I should have looked for that requirement, thanks. Any chance it'll be updated to support 12x, since thats the direction Comfy has gone?

Edit: Nevermind, I see that its onxx that is requiring it, so I assume you need to wait for them to add support.

Gourieff commented 9 months ago

Edit: Nevermind, I see that its onxx that is requiring it, so I assume you need to wait for them to add support.

Yeah, that's right

4DeepThought2 commented 9 months ago

@Gourieff and @ilcristofer I was able to get this all working with CUDA 12. Just needed to make wheel build of onxx using CUDA/NN 12.

git clone --recursive https://github.com/Microsoft/onnxruntime
cd onnxruntime
.\build.bat --update --build --use_cuda --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDNN\cudnn" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --skip_onnx_tests --build_wheel --parallel

Obvs change the paths to wherever you have cuda/cudnn

Its now working just fine in the latest version of ComfyUI I have a wheel file if you just want to pip install it @ilcristofer - feel free to inspect it first if safety is a concern. https://file.io/4RAvXi1dHVCK

Gourieff commented 9 months ago

@4DeepThought2 brilliant! 👍

nagaki09 commented 9 months ago

here is my case: after start server:

ComfyUI Revision: 1791 [248d9125] | Released on '2023-12-07'

Total VRAM 24576 MB, total RAM 32679 MB Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync VAE dtype: torch.bfloat16 Torch version: 2.1.1+cu121 Total VRAM 24576 MB, total RAM 32679 MB Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync VAE dtype: torch.bfloat16 Torch version: 2.1.1+cu121

I installed: ComfyUI_windows_portable_nvidia_cu121_or_cpu.7z

still met the same problem, any suggestion?

ilcristofer commented 9 months ago

@Gourieff and @ilcristofer I was able to get this all working with CUDA 12. Just needed to make wheel build of onxx using CUDA/NN 12.
git clone --recursive https://github.com/Microsoft/onnxruntime
cd onnxruntime
.\build.bat --update --build --use_cuda --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDNN\cudnn" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --skip_onnx_tests --build_wheel --parallel
Obvs change the paths to wherever you have cuda/cudnn

Its now working just fine in the latest version of ComfyUI I have a wheel file if you just want to pip install it @ilcristofer - feel free to inspect it first if safety is a concern. https://file.io/4RAvXi1dHVCK

@4DeepThought2 thanks for the support. I tried to do as you say and after the command .\build.bat --update --build --use_cuda --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\CUDDN\v8.9.7" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1" --skip_onnx_tests --build_wheel --parallel

I get this error 2023-12-07 23:45:57.006 build [ERROR] - Failed to resolve executable path for 'cmake'.

4DeepThought2 commented 9 months ago

You need to install cmake for windows: https://cmake.org/download/

michael-pptf commented 9 months ago

Yea, I gave up on this since downgrading to CUDA 11.8 would break so many other plugins.

Since it could work with CUDA 11.8, I am closing this issue.

ilcristofer commented 9 months ago

@Gourieff and @ilcristofer I was able to get this all working with CUDA 12. Just needed to make wheel build of onxx using CUDA/NN 12.
git clone --recursive https://github.com/Microsoft/onnxruntime
cd onnxruntime
.\build.bat --update --build --use_cuda --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDNN\cudnn" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --skip_onnx_tests --build_wheel --parallel
Obvs change the paths to wherever you have cuda/cudnn

Its now working just fine in the latest version of ComfyUI I have a wheel file if you just want to pip install it @ilcristofer - feel free to inspect it first if safety is a concern. https://file.io/4RAvXi1dHVCK

You need to install cmake for windows: https://cmake.org/download/

Hello. I tried as suggested and now it works, but not on the portable version. I did a manual installation of comfyui and it works (probably because it doesn't use the embedded Python version, but the standard one installed on the PC).

ilcristofer commented 9 months ago

I believe, however, that starting from this, it's possible to modify the node to work with the latest portable version of comfyui.

TheBill2001 commented 9 months ago

Can the node be set to use the CPU instead?

tetsuoo-online commented 8 months ago

I was about to open a new thread but it seems that I'm not the only one. So~~I'm pretty sure I had no more issues yesterday but here I am again. Now it's CUDA xD

I tried to follow the advices here but really I have no idea what I'm doing. I'm using ComfyUI portable (like, there is another version ?) and well, it didn't work.

Gourieff commented 8 months ago

https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements

ComfyUI has a lot of builds on their Releases page ORT-GPU requires CUDA 11.8 (this is official requirements) So, you should download latest ComfyUI with Cu118 https://github.com/comfyanonymous/ComfyUI/releases On some Machines it works fine with Cu121, but on some - not (I don't know what it depends on...) But ORT-GPU 100% works with Cu118

tetsuoo-online commented 8 months ago

Today Bing AI told me about Pinokio, from there I found out about FaceFusion which is okay, but not as good as your Reactor in ComfyUI (and it's censored) so~~ I have no other choice but to keep trying but to be honest, I'm tired of all that Python nightmare x) I already reinstalled ComfyUI and everything last time (like 2 days ago ?), I used the Download Link with stable pytorch 2.1 cu121 maybe that's how I end up with CUDA121 I don't know. I already read that onnxruntime doc page but it's very hardcore to put my head around this, anyway, should I install the unstable nightly build instead ? (isn't that unstable enough ? xDDD ) Or should I uninstall Cuda121 and put 118 instead (Cuda,Cdnn, maybe it's the same thing hell I don't know) I want to see the bottom of this

tetsuoo-online commented 8 months ago

okay case closed (for now), I installed ComfyUI using Pinokio, reinstalled all the custom nodes for the 4th time this week x) AND NOW IT WORKS AGAIN I don't get it but I don't care anymore x) (for now lol)

Symvor commented 8 months ago

Can the node be set to use the CPU instead?

If you want to force the node to use CPU you can go to ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py and then in line 29 change providers to ["CPUExecutionProvider"]. It's a lazy workaround but it works for me.

zhoobin021 commented 7 months ago

@Gourieff and @ilcristofer I was able to get this all working with CUDA 12. Just needed to make wheel build of onxx using CUDA/NN 12.
git clone --recursive https://github.com/Microsoft/onnxruntime
cd onnxruntime
.\build.bat --update --build --use_cuda --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDNN\cudnn" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --skip_onnx_tests --build_wheel --parallel
Obvs change the paths to wherever you have cuda/cudnn

Its now working just fine in the latest version of ComfyUI I have a wheel file if you just want to pip install it @ilcristofer - feel free to inspect it first if safety is a concern. https://file.io/4RAvXi1dHVCK

where should we "git clone --recursive https://github.com/Microsoft/onnxruntime" ? like what dir ?

I found 'onnxruntime' & 'onnxruntime_gpu-1.17.0.dist-info' folders in : C:\Users(name)\AppData\Local\Programs\Python\Python310\Lib\site-packages\onnxruntime

is that it?

QkiZMR commented 6 months ago

Can the node be set to use the CPU instead?

If you want to force the node to use CPU you can go to ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py and then in line 29 change providers to ["CPUExecutionProvider"]. It's a lazy workaround but it works for me.

It would be nice to have some settings like in A1111 Reactor. I can set CPU or GPU in A1111 to run Reactor there.

Kimononono commented 6 months ago

Ran into this issue running cuda 12.1 on linux. What fixed it for me was

pip install ort-nightly-gpu==1.17.0.dev20240118002 --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-12-nightly/pypi/simple/

which I found here

ProtoBelisarius commented 6 months ago

Can the node be set to use the CPU instead?

If you want to force the node to use CPU you can go to ComfyUI\custom_nodes\comfyui-reactor-node\scripts\reactor_swapper.py and then in line 29 change providers to ["CPUExecutionProvider"]. It's a lazy workaround but it works for me.

If on Linux and AMD(with RoCM) its also possible to just change the CUDAExecutionProvider to ROCMExecutionProvider and it works fine.

sebord commented 5 months ago

@4DeepThought2 It reported an error saying it couldn't be found, but I installed it.

alonsojr1980 commented 5 months ago

@Gourieff and @ilcristofer I was able to get this all working with CUDA 12. Just needed to make wheel build of onxx using CUDA/NN 12.
git clone --recursive https://github.com/Microsoft/onnxruntime
cd onnxruntime
.\build.bat --update --build --use_cuda --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDNN\cudnn" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --skip_onnx_tests --build_wheel --parallel
Obvs change the paths to wherever you have cuda/cudnn

Its now working just fine in the latest version of ComfyUI I have a wheel file if you just want to pip install it @ilcristofer - feel free to inspect it first if safety is a concern. https://file.io/4RAvXi1dHVCK

Please, be more detailed when posting. After compiling, what did you do? Where is the compiled file? What to do with it? Thanks.

maxwelhelp commented 1 month ago

To troubleshoot CUDA and ONNX Runtime issues:

Use Dependency Walker: Download and run Dependency Walker (http://www.dependencywalker.com/) to analyze onnxruntime_providers_cuda.dll. This will help identify missing dependencies. Check for missing files: The analysis revealed that the following files were missing:

cudnn64_8.dll cudnn_ops_infer64_8.dll cublasLt64_11.dll cublas64_11.dll

Locate the missing files: These files were found in the Python site-packages directory: AppData\Local\Programs\Python\Python39\Lib\site-packages Copy the files: The missing files were manually copied to the CUDA binary directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin Verify the solution: After copying the files, the CUDA-related issues with ONNX Runtime were resolved.

This manual file relocation solved the immediate problem by ensuring all required CUDA and cuDNN libraries were in a location accessible to the system and ONNX Runtime. However, for a more robust long-term solution, it's recommended to properly reinstall CUDA and cuDNN according to NVIDIA's official documentation to ensure all components are correctly placed and registered in the system.

AnjumShakeelJanjua commented 3 weeks ago

I have faced this error many times before but this video has solved the issue most of the time.

https://www.youtube.com/watch?v=Ywc5BUN5SSc&list=WL&index=1&t=10s

Gourieff / comfyui-reactor-node