search

microsoft / onnxruntime-genai

Generative AI extensions for onnxruntime
MIT License
497 stars 124 forks source link

Genny doesn't like Phi-3-mini-128k-instruct-onnx #938

Closed lilhoser closed 4 weeks ago

lilhoser commented 1 month ago

Describe the bug

The model will load but inference fails with a runtime error.

To Reproduce Steps to reproduce the behavior:

  1. Download any model in https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx
  2. For example, select the CPU model at Phi-3-mini-128k-instruct-onnx\cpu_and_mobile\cpu-int4-rtn-block-32-acc-level-4
  3. Run Genny, choose this model folder
  4. See error

Expected behavior

It should support phi 3?

Screenshots

image

Desktop (please complete the following information):

natke commented 4 weeks ago

Hi @lilhoser, which version of ONNX Runtime GenAI are you running? And which version of the example? I cannot reproduce your issue

lilhoser commented 4 weeks ago

@natke thanks for the ping. I can't seem to repro this either. Perhaps I had mismatched runtimes with model type. Will close for now.