[Bug]: ONNX Runtime Fails to Load CUDA Provider

Checklist

[ ] The issue exists after disabling all extensions
[X] The issue exists on a clean installation of webui
[X] The issue is caused by an extension, but I believe it is caused by a bug in the webui
[X] The issue exists in the current version of the webui
[x] The issue has not been reported before recently
[X] The issue has been reported before but has not been fixed yet

What happened?

I am experiencing issues with ONNX Runtime failing to load the CUDA provider. Despite setting up the environment correctly, ONNX Runtime is unable to utilize the GPU, resulting in the following error message:

RuntimeError: D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:857 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded.
Please install the correct version of CUDA and cuDNN as mentioned in the GPU requirements page, make sure they're in the PATH, and that your GPU is supported.

Steps to reproduce the problem

Install CUDA 12.4 and cuDNN 9.1.1.
Set up the environment variables:

$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin"
$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\libnvvp"
$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\extras\CUPTI\lib64"
$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"

Activate the virtual environment

Uninstall and reinstall ONNX Runtime with GPU support:

pip uninstall onnxruntime-gpu
pip install onnxruntime-gpu

Run a test script to load an ONNX model with CUDA support:
```
import onnxruntime as ort
import logging
```

ort.set_default_logger_severity(0) # Set to 0 for detailed logging

try: providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] model_path = 'C:/Users/Insider/Downloads/squeezenet.onnx' session = ort.InferenceSession(model_path, providers=providers) print("Providers:", session.get_providers()) except Exception as e: logging.error("Error during ONNX Runtime session creation:", exc_info=e)


### What should have happened?

The ONNX Runtime should successfully load the CUDA provider and use the GPU for inference without any errors.

Reactor, Rembg, etc have not worked and present this issue.

### What browsers do you use to access the UI ?

Google Chrome, Other

### Sysinfo

[sysinfo-2024-05-25-07-48.json](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/15442178/sysinfo-2024-05-25-07-48.json)

### Console logs

Shell Already up to date. venv "D:\ImagegenProject\sd.webui\webui\venv\Scripts\Python.exe" Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Version: v1.9.3 Commit hash: 1c0a0c4c26f78c32095ebc7f8af82f5c04fca8c0 CUDA 12.1 ####################################################################################################### Initializing Civitai Link If submitting an issue on github, please provide the below text for debugging purposes:

Python revision: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Civitai Link revision: 115cd9c35b0774c90cb9c397ad60ef6a7dac60de SD-WebUI revision: 1c0a0c4c26f78c32095ebc7f8af82f5c04fca8c0

Checking Civitai Link requirements...

####################################################################################################### Launching Web UI with arguments: --autolaunch --upcast-sampling --xformers --no-half-vae --theme=dark Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu. [-] ADetailer initialized. version: 24.4.2, num models: 15 *** Error loading script: main.py Traceback (most recent call last): File "D:\ImagegenProject\sd.webui\webui\modules\scripts.py", line 508, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "D:\ImagegenProject\sd.webui\webui\modules\script_loading.py", line 13, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "D:\ImagegenProject\sd.webui\webui\extensions\openpose-editor\scripts\main.py", line 14, in from basicsr.utils.download_util import load_file_from_url ModuleNotFoundError: No module named 'basicsr'

[AddNet] Updating model hashes... 0it [00:00, ?it/s] [AddNet] Updating model hashes... 0it [00:00, ?it/s] ControlNet preprocessor location: D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-controlnet\annotator\downloads 2024-05-25 03:40:22,282 - ControlNet - INFO - ControlNet v1.1.448 [sdwi2iextender] Developper warning: [sdwi2iextender] ./modules/img2img.py is being recompiled at run time with a patch. Your debugger will not work in this file. [sdwi2iextender] If you need debug tools in this file, disable all extensions that use the sdwi2iextender library. [sdwi2iextender] This patch is temporary and will be removed when v1.9 will be released. 03:40:23 - ReActor - STATUS - Running v0.7.0-b7 on Device: CUDA Civitai: API loaded Loading weights [821aa5537f] from D:\ImagegenProject\sd.webui\webui\models\Stable-diffusion\autismmixSDXL_autismmixPony.safetensors 2024-05-25 03:40:23,697 - ControlNet - INFO - ControlNet UI callback registered. Creating model from config: D:\ImagegenProject\sd.webui\webui\repositories\generative-models\configs\inference\sd_xl_base.yaml Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). 🤯 LobeTheme: Initializing... Civitai: Check resources for missing info files Civitai: Check resources for missing preview images Startup time: 15.2s (prepare environment: 6.0s, import torch: 2.7s, import gradio: 0.6s, setup paths: 0.7s, initialize shared: 0.1s, other imports: 0.3s, list SD models: 0.4s, load scripts: 2.8s, create ui: 1.3s, gradio launch: 0.2s). Civitai: Found 6 resources missing preview images Civitai: Found 5 resources missing info files Civitai: Found 1 hash matches Exception in thread Thread-49 (load_previews): Traceback (most recent call last): File "C:\Users\Insider\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner self.run() File "C:\Users\Insider\AppData\Local\Programs\Python\Python310\lib\threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "D:\ImagegenProject\sd.webui\webui\extensions\sd_civitai_extension\scripts\previews.py", line 50, in load_previews if (nsfw_previews is False): images = [i for i in images if i['nsfw'] is False] File "D:\ImagegenProject\sd.webui\webui\extensions\sd_civitai_extension\scripts\previews.py", line 50, in if (nsfw_previews is False): images = [i for i in images if i['nsfw'] is False] KeyError: 'nsfw' Civitai: No info found on Civitai Loading VAE weights specified in settings: D:\ImagegenProject\sd.webui\webui\models\VAE\fixFP16ErrorsSDXLLowerMemoryUse_v10.safetensors Applying attention optimization: xformers... done. Model loaded in 7.5s (load weights from disk: 0.4s, create model: 0.4s, apply weights to model: 3.5s, load VAE: 0.1s, load textual inversion embeddings: 2.8s, calculate empty prompt: 0.2s). 03:50:22 - ReActor - STATUS - Working: source face index [0], target face index [0] 03:50:22 - ReActor - STATUS - Analyzing Source Image... 2024-05-25 03:50:22.9010136 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

EP Error EP Error C:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported. when using ['CUDAExecutionProvider'] Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.

2024-05-25 03:50:22.9820565 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Error completing request Arguments: ('task(ej3p1nbym2wmliz)', 0.0, <PIL.Image.Image image mode=RGBA size=1010x1263 at 0x23F748617E0>, None, '', '', True, True, 0.0, 1, 0.0, 512, 512, True, 'None', 'None', 0, False, 1, False, 1, 0, False, 0.5, 0.2, False, 0.9, 0.15, 0.5, False, False, 384, 768, 4096, 409600, 'Maximize area', 0.1, False, ['Horizontal'], False, ['Deepbooru'], <PIL.Image.Image image mode=RGB size=947x1263 at 0x23F748634C0>, True, '0', '0', 'inswapper_128.onnx', 'CodeFormer', 1, True, 'None', 1, 1, 1, 0, 0, 0.5, 'CUDA', False, 0, 'None', '', None, False, False, 0.5, 0, False, 'None', False, False, 240, 10, 10) {} Traceback (most recent call last): File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) RuntimeError: C:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\ImagegenProject\sd.webui\webui\modules\call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "D:\ImagegenProject\sd.webui\webui\modules\call_queue.py", line 36, in f
    res = func(*args, **kwargs)
  File "D:\ImagegenProject\sd.webui\webui\modules\postprocessing.py", line 131, in run_postprocessing_webui
    return run_postprocessing(*args, **kwargs)
  File "D:\ImagegenProject\sd.webui\webui\modules\postprocessing.py", line 71, in run_postprocessing
    scripts.scripts_postproc.run(initial_pp, args)
  File "D:\ImagegenProject\sd.webui\webui\modules\scripts_postprocessing.py", line 198, in run
    script.process(single_image, **process_args)
  File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_faceswap.py", line 663, in process
    result, output, swapped = swap_face(
  File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_swapper.py", line 544, in swap_face
    source_faces = analyze_faces(source_img, det_thresh=detection_options.det_thresh, det_maxnum=detection_options.det_maxnum)
  File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_swapper.py", line 302, in analyze_faces
    face_analyser = copy.deepcopy(getAnalysisModel())
  File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_swapper.py", line 145, in getAnalysisModel
    ANALYSIS_MODEL = insightface.app.FaceAnalysis(
  File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\console_log_patch.py", line 48, in patched_faceanalysis_init
    model = model_zoo.get_model(onnx_file, **kwargs)
  File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 96, in get_model
    model = router.get_model(providers=providers, provider_options=provider_options)
  File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\console_log_patch.py", line 21, in patched_get_model
    session = PickableInferenceSession(self.onnx_file, **kwargs)
  File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 25, in __init__
    super().__init__(model_path, **kwargs)
  File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 432, in __init__
    raise fallback_error from e
  File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 427, in __init__
    self._create_inference_session(self._fallback_providers, None)
  File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.


### Additional information

- I have installed the compatible versions of CUDA (12.4) and cuDNN (9.1.1).
- My GPU drivers are up to date.
- I am running an NVIDIA RTX 3080.

The environment variables are set as follows:
```$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin"
$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\libnvvp"
$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\extras\CUPTI\lib64"
$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"

I have reinstalled ONNX Runtime with GPU support:

First, this is an issue that should be reported to https://github.com/Gourieff/sd-webui-reactor.

I don't understand why, this issue occurs in more instances than the one while using reactor.

Second, according to onnxruntime's documentation, the onnxruntime-gpu package that you install with pip expects CUDA version 11.8.

Deleting the venv folder from my SD installation and taking note of what it did when it rebuilt says that it is using CUDA 12.1, despite myself having since uninstalled 12.4, and replacing it with 11.8 hoping for a drop in, very easy, fix.

2024-05-25 17:31:00.0056574 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

*************** EP Error ***************
EP Error C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.
 when using ['CUDAExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
****************************************
2024-05-25 17:31:00.1128273 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

*** Error completing request
*** Arguments: ('task(kedi2ywxxrunhi4)', 0.0, <PIL.Image.Image image mode=RGBA size=3072x4096 at 0x24A9DCDBE80>, None, '', '', True, False, 0.0, 4, 0.0, 512, 512, True, 'None', 'None', 0, False, 1, False, 1, 0, False, 0.5, 0.2, False, 0.9, 0.15, 0.5, False, False, 384, 768, 4096, 409600, 'Maximize area', 0.1, False, ['Horizontal'], False, ['Deepbooru'], <PIL.Image.Image image mode=RGB size=1536x2048 at 0x24A9DCDBDC0>, True, '0', '0', 'inswapper_128.onnx', 'CodeFormer', 1, True, 'None', 1, 1, 1, 0, 0, 0.5, 'CUDA', False, 0, 'None', '', None, False, False, 0.5, 0, False, 'None', False, False, 240, 10, 10) {}
    Traceback (most recent call last):
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in __init__
        self._create_inference_session(providers, provider_options, disabled_optimizers)
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session
        sess.initialize_session(providers, provider_options, disabled_optimizers)
    RuntimeError: C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "D:\ImagegenProject\sd.webui\webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "D:\ImagegenProject\sd.webui\webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "D:\ImagegenProject\sd.webui\webui\modules\postprocessing.py", line 131, in run_postprocessing_webui
        return run_postprocessing(*args, **kwargs)
      File "D:\ImagegenProject\sd.webui\webui\modules\postprocessing.py", line 71, in run_postprocessing
        scripts.scripts_postproc.run(initial_pp, args)
      File "D:\ImagegenProject\sd.webui\webui\modules\scripts_postprocessing.py", line 198, in run
        script.process(single_image, **process_args)
      File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_faceswap.py", line 663, in process
        result, output, swapped = swap_face(
      File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_swapper.py", line 544, in swap_face
        source_faces = analyze_faces(source_img, det_thresh=detection_options.det_thresh, det_maxnum=detection_options.det_maxnum)
      File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_swapper.py", line 302, in analyze_faces
        face_analyser = copy.deepcopy(getAnalysisModel())
      File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\reactor_swapper.py", line 145, in getAnalysisModel
        ANALYSIS_MODEL = insightface.app.FaceAnalysis(
      File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\console_log_patch.py", line 48, in patched_faceanalysis_init
        model = model_zoo.get_model(onnx_file, **kwargs)
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 96, in get_model
        model = router.get_model(providers=providers, provider_options=provider_options)
      File "D:\ImagegenProject\sd.webui\webui\extensions\sd-webui-reactor\scripts\console_log_patch.py", line 21, in patched_get_model
        session = PickableInferenceSession(self.onnx_file, **kwargs)
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 25, in __init__
        super().__init__(model_path, **kwargs)
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 432, in __init__
        raise fallback_error from e
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 427, in __init__
        self._create_inference_session(self._fallback_providers, None)
      File "D:\ImagegenProject\sd.webui\webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session
        sess.initialize_session(providers, provider_options, disabled_optimizers)
    RuntimeError: C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.

AUTOMATIC1111 / stable-diffusion-webui