search

Gourieff / sd-webui-reactor

Fast and Simple Face Swap Extension for StableDiffusion WebUI (A1111 SD WebUI, SD WebUI Forge, SD.Next, Cagliostro)
GNU Affero General Public License v3.0
2.56k stars 279 forks source link

[ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory #483

Open Amit30swgoh opened 4 months ago

Amit30swgoh commented 4 months ago

First, confirm

What happened?

image image [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

please help me because i want to run it faster useing GPU

Steps to reproduce the problem

when running Reactor fast face getting ONNXRuntimeError Error :( please help friend.

Sysinfo

google colab pro

Relevant console log

FETCH DATA from: /content/drive/MyDrive/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json [DONE]
#clip_model_path chatml None
#model 
start_local_llm error
#read_workflow_json_files_all /content/drive/MyDrive/ComfyUI/custom_nodes/comfyui-mixlab-nodes/app/
got prompt
[rgthree] Using rgthree's optimized recursive execution.
[rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute.
[rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI.
[ReActor] 12:36:01 - STATUS - Working: source face index [0], target face index [0]
[ReActor] 12:36:02 - STATUS - Analyzing Source Image...
2024-07-18 12:36:02.316082451 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.004577726 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.059246241 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.100719245 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.411743966 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

[ReActor] 12:36:04 - STATUS - Analyzing Target Image 0...
[ReActor] 12:36:04 - STATUS - Analyzing Target Image 1...
[ReActor] 12:36:05 - STATUS - Analyzing Target Image 2...
[ReActor] 12:36:05 - STATUS - Analyzing Target Image 3...
[ReActor] 12:36:05 - STATUS - Analyzing Target Image 4...
[ReActor] 12:36:06 - STATUS - Analyzing Target Image 5...
[ReActor] 12:36:06 - STATUS - Analyzing Target Image 6...
[ReActor] 12:36:06 - STATUS - Analyzing Target Image 7...
[ReActor] 12:36:07 - STATUS - Analyzing Target Image 8...
[ReActor] 12:36:07 - STATUS - Analyzing Target Image 9...
[ReActor] 12:36:07 - STATUS - Analyzing Target Image 10...
[ReActor] 12:36:08 - STATUS - Analyzing Target Image 11...
[ReActor] 12:36:08 - STATUS - Analyzing Target Image 12...
[ReActor] 12:36:08 - STATUS - Analyzing Target Image 13...
[ReActor] 12:36:09 - STATUS - Analyzing Target Image 14...
[ReActor] 12:36:09 - STATUS - Analyzing Target Image 15...
[ReActor] 12:36:09 - STATUS - Analyzing Target Image 16...
[ReActor] 12:36:10 - STATUS - Analyzing Target Image 17...
[ReActor] 12:36:10 - STATUS - Analyzing Target Image 18...
[ReActor] 12:36:10 - STATUS - Analyzing Target Image 19...
[ReActor] 12:36:11 - STATUS - Analyzing Target Image 20...
[ReActor] 12:36:11 - STATUS - Analyzing Target Image 21...
[ReActor] 12:36:11 - STATUS - Analyzing Target Image 22...
[ReActor] 12:36:12 - STATUS - Analyzing Target Image 23...
[ReActor] 12:36:12 - STATUS - Analyzing Target Image 24...
[ReActor] 12:36:13 - STATUS - Analyzing Target Image 25...
[ReActor] 12:36:13 - STATUS - Analyzing Target Image 26...
[ReActor] 12:36:13 - STATUS - Analyzing Target Image 27...
[ReActor] 12:36:14 - STATUS - Analyzing Target Image 28...
[ReActor] 12:36:14 - STATUS - Analyzing Target Image 29...
[ReActor] 12:36:15 - STATUS - Analyzing Target Image 30...
[ReActor] 12:36:15 - STATUS - Analyzing Target Image 31...
[ReActor] 12:36:15 - STATUS - Analyzing Target Image 32...
[ReActor] 12:36:16 - STATUS - Analyzing Target Image 33...
[ReActor] 12:36:16 - STATUS - Analyzing Target Image 34...
[ReActor] 12:36:17 - STATUS - Analyzing Target Image 35...
[ReActor] 12:36:17 - STATUS - Analyzing Target Image 36...
[ReActor] 12:36:17 - STATUS - Analyzing Target Image 37...
[ReActor] 12:36:18 - STATUS - Analyzing Target Image 38...
[ReActor] 12:36:18 - STATUS - Analyzing Target Image 39...
[ReActor] 12:36:18 - STATUS - Analyzing Target Image 40...
[ReActor] 12:36:19 - STATUS - Analyzing Target Image 41...
[ReActor] 12:36:19 - STATUS - Analyzing Target Image 42...
[ReActor] 12:36:19 - STATUS - Analyzing Target Image 43...
[ReActor] 12:36:20 - STATUS - Analyzing Target Image 44...
[ReActor] 12:36:20 - STATUS - Analyzing Target Image 45...
[ReActor] 12:36:20 - STATUS - Analyzing Target Image 46...
[ReActor] 12:36:21 - STATUS - Source Face 0: Detected Gender -F-
[ReActor] 12:36:21 - STATUS - OK - Detected Gender matches Condition
2024-07-18 12:36:22.077235867 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

[ReActor] 12:36:25 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:25 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:25 - STATUS - Swapping 0...
[ReActor] 12:36:26 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:26 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:26 - STATUS - Swapping 1...
[ReActor] 12:36:28 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:28 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:28 - STATUS - Swapping 2...
[ReActor] 12:36:30 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:30 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:30 - STATUS - Swapping 3...
[ReActor] 12:36:31 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:31 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:31 - STATUS - Swapping 4...
[ReActor] 12:36:33 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:33 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:33 - STATUS - Swapping 5...
[ReActor] 12:36:34 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:34 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:34 - STATUS - Swapping 6...
[ReActor] 12:36:36 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:36 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:36 - STATUS - Swapping 7...
[ReActor] 12:36:38 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:38 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:38 - STATUS - Swapping 8...
[ReActor] 12:36:39 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:39 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:39 - STATUS - Swapping 9...
[ReActor] 12:36:39 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:39 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:39 - STATUS - Swapping 10...
[ReActor] 12:36:40 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:40 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:40 - STATUS - Swapping 11...
[ReActor] 12:36:41 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:41 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:41 - STATUS - Swapping 12...
[ReActor] 12:36:42 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:42 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:42 - STATUS - Swapping 13...
[ReActor] 12:36:44 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:44 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:44 - STATUS - Swapping 14...
[ReActor] 12:36:45 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:45 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:45 - STATUS - Swapping 15...
[ReActor] 12:36:47 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:47 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:47 - STATUS - Swapping 16...
[ReActor] 12:36:48 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:48 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:48 - STATUS - Swapping 17...
[ReActor] 12:36:49 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:49 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:49 - STATUS - Swapping 18...
[ReActor] 12:36:50 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:50 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:50 - STATUS - Swapping 19...
[ReActor] 12:36:50 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:50 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:50 - STATUS - Swapping 20...
[ReActor] 12:36:51 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:51 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:51 - STATUS - Swapping 21...
[ReActor] 12:36:52 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:52 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:52 - STATUS - Swapping 22...
[ReActor] 12:36:53 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:53 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:53 - STATUS - Swapping 23...
[ReActor] 12:36:54 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:54 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:54 - STATUS - Swapping 24...
[ReActor] 12:36:55 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:55 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:55 - STATUS - Swapping 25...
[ReActor] 12:36:56 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:56 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:56 - STATUS - Swapping 26...
[ReActor] 12:36:56 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:56 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:56 - STATUS - Swapping 27...
[ReActor] 12:36:58 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:58 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:58 - STATUS - Swapping 28...
[ReActor] 12:36:59 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:59 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:59 - STATUS - Swapping 29...
[ReActor] 12:37:01 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:01 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:01 - STATUS - Swapping 30...
[ReActor] 12:37:03 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:03 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:03 - STATUS - Swapping 31...
[ReActor] 12:37:04 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:04 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:04 - STATUS - Swapping 32...
[ReActor] 12:37:06 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:06 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:06 - STATUS - Swapping 33...
[ReActor] 12:37:07 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:07 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:07 - STATUS - Swapping 34...
[ReActor] 12:37:08 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:08 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:08 - STATUS - Swapping 35...
[ReActor] 12:37:09 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:09 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:09 - STATUS - Swapping 36...
[ReActor] 12:37:10 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:10 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:10 - STATUS - Swapping 37...
[ReActor] 12:37:11 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:11 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:11 - STATUS - Swapping 38...
[ReActor] 12:37:12 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:12 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:12 - STATUS - Swapping 39...
[ReActor] 12:37:14 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:14 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:14 - STATUS - Swapping 40...
[ReActor] 12:37:15 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:15 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:15 - STATUS - Swapping 41...
[ReActor] 12:37:17 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:17 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:17 - STATUS - Swapping 42...
[ReActor] 12:37:19 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:19 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:19 - STATUS - Swapping 43...
[ReActor] 12:37:20 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:20 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:20 - STATUS - Swapping 44...
[ReActor] 12:37:21 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:21 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:21 - STATUS - Swapping 45...
[ReActor] 12:37:22 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:22 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:22 - STATUS - Swapping 46...
[ReActor] 12:37:23 - STATUS - --Done!--
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Final clearing cache... Done cache clearing

Additional information

No response

Amit30swgoh commented 4 months ago

please help @Gourieff perhaps its because of onnxruntime-gpu 1.18.1? perhaps need to change to 1.15? or move back to "insightface==0.7.3 onnx>=1.14.0 opencv-python>=4.7.0.72 numpy segment_anything" ?

Amit30swgoh commented 4 months ago

managed to fix but it doesnt utilize vram so 10 sec vid takes 40 min :( image

andreclaudino commented 3 months ago

I have this exactly same problem on ComfyUI start. Tyried to downgrade packages as you recommended but didn't work.

aifantasyglam commented 2 months ago

same problem using Colab Pro, A1111, SD1.5, but just with the first image

image

aifantasyglam commented 2 months ago

@Gourieff Hi, could you help us please...?!

Gourieff commented 1 month ago

What version of CUDA do you use guys?

Ken-g6 commented 1 month ago

I'm getting this error message that looks related to this issue:

2024-09-29 22:41:19.252909826 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

If I locate libcublasLt.so, I see that A1111 has installed stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/lib/libcublasLt.so.12. I also happen to have libcublasLt.so.10 in my system. But not libcublasLt.so.11.