Closed Bagashidayat1 closed 9 months ago
I believe it is not supporting CUDA 12, I have tried 12.2 and getting below error.
'[ErrorCode:RuntimeException] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load
2023-12-19 07:23:15.717360679 [E:onnxruntime:Default, provider_bridge_ort.cc:1498 TryGetProviderInfo_CUDA]
/onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1195 onnxruntime::Provider&
onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library
libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
Using the nightly builds via
pip install ort-nightly-gpu==1.17.0.dev20231215001 --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple
Does anyone know how many days need to support CUDA12?
Does anyone know how many days need to support CUDA12?
no idea
current version 1.16.x support cuda 12, but must build it from source
Using the nightly builds via pip install ort-nightly-gpu==1.17.0.dev20231215001 --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple
In my case it didn't help to use the latest nightly (...002). It's still looking for cuda11/libcublasLt.so.11 while cuda12 has libcublasLt.so.12. I'm already trying to build from source for 2 days but with Google Collab limitations that seems like an impossible endeavor.
[E:onnxruntime:Default, provider_bridge_ort.cc:1498 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1195 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
here my instructions to build onnxruntime with cuda 12 on colab
!apt-get install tensorrt-dev
!git clone \
--branch=rel-1.16.3 \
--single-branch \
--depth=1 \
--recurse-submodules \
--shallow-submodules \
https://github.com/microsoft/onnxruntime
%cd onnxruntime
!python tools/ci_build/build.py \
--build_dir=build \
--build \
--update \
--parallel \
--skip_tests \
--build_wheel \
--enable_pybind \
--config=Release \
--skip_submodule_sync \
--allow_running_as_root \
--compile_no_warning_as_error \
--use_cuda \
--use_tensorrt \
--cuda_home=/usr/local/cuda \
--cudnn_home=/usr/lib/x86_64-linux-gnu \
--tensorrt_home=/usr/lib/x86_64-linux-gnu \
--cmake_extra_defines='CMAKE_CUDA_ARCHITECTURES=75' \
--numpy_version="$(python -c 'import numpy; print(numpy.__version__)')"
take ~2h, no need gpu session (remove tensorrt if not needed)
wheel file created: onnxruntime/build/Release/dist/onnxruntime_gpu-1.16.3-cp310-cp310-linux_x86_64.whl
save it elsewhere to use in gpu session
easier solution but only temporary: https://github.com/googlecolab/colabtools/issues/4214#issuecomment-1859155789
For people who will come after me, you can download whl here https://dev.azure.com/onnxruntime/onnxruntime/_artifacts/feed/onnxruntime-cuda-12/PyPI/onnxruntime-gpu/overview/1.17.0
For people who will come after me, you can download whl here https://dev.azure.com/onnxruntime/onnxruntime/_artifacts/feed/onnxruntime-cuda-12/PyPI/onnxruntime-gpu/overview/1.17.0
Thank you! Alternatively, you can install the package via pip:
pip install onnxruntime-gpu==1.17.0 --index-url=https://pkgs.dev.azure.com/onnxruntime/onnxruntime/_packaging/onnxruntime-cuda-12/pypi/simple
Previously, I used a slightly different package, maybe someone will find it useful too:
pip install ort-nightly-gpu==1.17.0.dev20231205004 --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-12-nightly/pypi/simple/
Nice to see you Oleksandr.
What is the actual difference between both packages? I assume the one on azure is from the onnxruntime team and based on the latest build. The other by one of the onnxruntime devs and locked to a version?
Using a locked version feels more suitable.
Nice to see you Oleksandr.
What is the actual difference between both packages? I assume the one on azure is from the onnxruntime team and based on the latest build. The other by one of the onnxruntime devs and locked to a version?
Using a locked version feels more suitable.
Hi Henry, good to see you too. In my case, at the moment both packages work in Colab with Cuda 12, so it seems to me that there is no big difference which one to use. At least for now there are no critical bugs in the package based on the latest build.
Thank you so much for addressing and fixing the existing defects in onnxruntime-gpu. Your efforts are truly appreciated!
So the nightly onnxruntime-gpu (v 1.17) should work? @netrunner-exe Oleksandr, what CUDA and cuDNN versions (12.? and 8.9.6?) did you manage to compile with?
So the nightly onnxruntime-gpu (v 1.17) should work? @netrunner-exe Oleksandr, what CUDA and cuDNN versions (12.? and 8.9.6?) did you manage to compile with?
@AndreyOrb, I tested it on Ubuntu 22.04.3 LTS and CUDA 12.2 (Google Colab). @henryruhs Henry, after small tests of both packages, it turned out that Codeformer converted to .onnx does not work with pip install onnxruntime-gpu --index-url=https://pkgs.dev.azure.com/onnxruntime/onnxruntime/_packaging/onnxruntime-cuda-12/pypi/simple
. The second one - works.
------SYSTEM INFO------
Platform: Linux-6.1.58+-x86_64-with-glibc2.35
Chainer: 7.8.1
ChainerX: Not Available
NumPy: 1.23.5
CuPy:
OS : Linux-6.1.58+-x86_64-with-glibc2.35
Python Version : 3.10.12
CuPy Version : 12.2.0
CuPy Platform : NVIDIA CUDA
NumPy Version : 1.23.5
SciPy Version : 1.11.4
Cython Build Version : 0.29.36
Cython Runtime Version : 3.0.7
CUDA Root : /usr/local/cuda
nvcc PATH : /usr/local/cuda/bin/nvcc
CUDA Build Version : 12020
CUDA Driver Version : 12020
CUDA Runtime Version : 12020
cuBLAS Version : 120205
cuFFT Version : 11008
cuRAND Version : 10303
cuSOLVER Version : (11, 5, 2)
cuSPARSE Version : 12102
NVRTC Version : (12, 2)
Thrust Version : 200101
CUB Build Version : 200101
Jitify Build Version : <unknown>
cuDNN Build Version : 8801
cuDNN Version : 8906
NCCL Build Version : 21602
NCCL Runtime Version : 21903
cuTENSOR Version : None
cuSPARSELt Build Version : None
Device 0 Name : Tesla T4
Device 0 Compute Capability : 75
Device 0 PCI Bus ID : 0000:00:04.0
iDeep: Not Available
@netrunner-exe I can report have issues with occluder.onnx
on google colab using ort-nightly-gpu==1.17.0.dev20231205004
.
@netrunner-exe Thanks for sharing the details. Windows version has compiled and worked with my ONNX graphs with this command: .\build.bat --config Release --build_shared_lib --parallel --use_cuda --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\cuDNN\cudnn-windows-x86_64-8.9.6.50_cuda12-archive" --cuda_version 12.2 --skip_tests --use_mimalloc --use_tensorrt --tensorrt_home "C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT\TensorRT-8.6.1.6.Windows10.x86_64.cuda-12.0\TensorRT-8.6.1.6" --use_tensorrt_oss_parser
The CUDA 12 support will be official in upcoming Onnxrtime 1.17 realase, in a week. Official instruction on how to download it will be provided then.
Any updates? (Looking for support 12.3 on local machine(ubuntu))
solution from https://onnxruntime.ai/docs/install/
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
Describe the feature request
Is ONNXRUNTIME support CUDA 12 ?
Describe scenario use case
i got error on google colab using onnxruntime but it doesnt want to use T4 GPU