Open MajorPainTheCactus opened 1 month ago
@kunal-vaishnavi do let me know if you need any help reproducing. I can provide memory dumps etc. I want to distribute Phi3 with my application so this is very important to me.
Can you try building both ONNX Runtime and ONNX Runtime GenAI from source and running the example again to see if the error goes away? There have been multiple DirectML fixes that have been merged into both recently.
MajorPainTheCactus I just tried running the example on my end and didn't see any problem running with DML.
I updated the example script to make it simpler to use with the phi3 language model in this PR: https://github.com/microsoft/onnxruntime-genai/pull/607. Could you please give it a try?
-------------
Hello, Phi-3!
-------------
C API
Creating model...
Creating tokenizer...
Prompt:
Tell me a joke on Pirates.
Generating response...
Why don't pirates like to go to the doctor?
Because they always fear the "medical" bills!
(Note: While this joke is light-hearted, it's essential to remember that jokes about pirates or any other group should be respectful and not perpetuate stereotypes.)
Also, could you try the command line following the instructions here: https://github.com/microsoft/onnxruntime-genai/blob/51688ae75f7060d2db52e48ddd6970b1ee9d0d13/examples/c/README.md#build-the-cmake-project instead of cmake GUI?
@baijumeswani Thank you ever so much for looking into this. I'll update, follow those instructions and see where it gets me and let you know. Thanks again!
Hi so I've just built the latest version of onnxruntime-genai c example which created a phi3.exe by VS2022 that using cmake 3.26.3 excludes CUDA and uses DirectML. When I run this with various phi 3 models downloaded from hugging face:
https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/tree/main (directml folder) Debug command line: C:\Development\Phi-3-mini-4k-instruct-onnx\directml\directml-int4-awq-block-128
https://huggingface.co/microsoft/Phi-3-medium-4k-instruct-onnx-directml/tree/main Debug command line: C:\Development\Phi-3-medium-128k-instruct-onnx-directml\directml-int4-awq-block-128
I get a similar error:
Text Debug Output in VS2022:
With the executable outputting this:
Screenshot:![image](https://github.com/microsoft/onnxruntime-genai/assets/89490246/a5f44a75-75d7-42f9-9893-9b0dcfc42faa)
Text Output:
Any idea what I might be doing wrong?
Many thanks!
My exe directory:
DebugDir.zip
cmake configured options:
![image](https://github.com/microsoft/onnxruntime-genai/assets/89490246/4161821b-b7e1-4488-9687-9343ed347bb5)