Unable to install and run Phiv3 vision

[billSP2]

I've seen several others have issues installing phiv3 vision, but after following the instructions I was unsuccessful so decided to post my situation to solicit guidance.

Running on Windows 11 Laptop with Geforce RTX 3050 Ti Laptop GPU

Nvidia site says it supports CUDA. I installed CUDA via windows installer

I installed CUDNN via pip install. List of installed packages provided in the pip list output command attached. list.txt

Validated the following CUDA_PATH

and PATH are set

I installed the genai cuda libraries pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/

I have installed the model cuda-fp16

Try running phi3v.py and receive the onnxruntime_genai module could not be found error. python phi3v.py -m "C:\cuda-fp16\phi-3-v-128k-instruct-vision.onnx" Traceback (most recent call last): File "C:\Python312\Lib\site-packages\onnxruntime_genai__init__.py", line 10, in from onnxruntime_genai.onnxruntime_genai import * ImportError: DLL load failed while importing onnxruntime_genai: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\BillBaer\OneDrive - Support Partners\projects\cobra\GPT4Video\phi3\phi3v.py", line 12, in import onnxruntime_genai as og File "C:\Python312\Lib\site-packages\onnxruntime_genai__init__.py", line 18, in from onnxruntime_genai.onnxruntime_genai import * ImportError: DLL load failed while importing onnxruntime_genai: The specified module could not be found.

I went to the python site packages directory for onnxruntime_genai and I don't see the onnxruntime_genai.dll file. I've tried reinstalling the onnxruntime_genai_cuda multiple times but it does not seem to install that dll.

Appreciate your help in guiding me on what I'm doing wrong?

[baijumeswani]

The problem seems to be the version of cuda you are using. onnxruntime-genai-cuda requires cuda 11.8 and cudnn 8.x for cuda 11.8.

If you would like to use cuda 12, you can install using:

pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ onnxruntime-genai-cuda

[billSP2]

Thank you, bajiumeswani. I went with your suggestion and went back to 11.8 of cuda and 8.x for cudnn. Here is validation of what I have. I first uninstalled cuda 12 and deleted the cudnn files from that install making it fresh.

I copied the files in this archive into the bin, lib and include folders.

I reinstalled the models into c:\cuda-fp16.

Download just the FP16 model

$ huggingface-cli download microsoft/Phi-3-vision-128k-instruct-onnx-cuda --include cuda-fp16/* --local-dir . --local-dir-use-symlinks False

I reinstalled onnxruntime pip install --pre onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/

Cleaned out the cuda packages installed in pip. Here is my latest list. pip list.txt

Updated my path

Reinstalled the C++ redistributables just in case.

Wrote a small program to test the onnx package version import onnxruntime_genai print(f"onnxruntime version: {onnxruntime_genai.version}") Here is the output

Tried running phi3v.py again and received load error 126 which looks to have gotten further into the code.

Here is my onnxruntime_genai folder.

Any ideas what this new problem might be?

[baijumeswani]

onnxruntime requires cudnn 8.x for cuda 11.8. Download link: https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/11.x/cudnn-windows-x86_64-8.9.7.29_cuda11-archive.zip/

[billSP2]

baijumeswani, thank you for all of your guidance and sorry I still had the wrong cudnn. I had grabbed the most recent archived version for that version, but apparently chose wrong. The script now works so thank you again. All the best. Bill

[ricoisme]

I installed cuda 11.8 and cudnn 8.9.7.29 on windows11 but still not work.

update: I sloved my problem, just removed Microsoft.ML.OnnxRuntimeGenAI then back to work.