Closed AshD closed 3 months ago
Hi @AshD We are looking into this
@AshD what version of onnxruntime-genai are you using?
Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.2.0-rc4 in C# when running.
Latest onnxruntime-genai installed from pip for the python onnxruntime_genai.models.builder command
The genai_config.json
file shows a list of pad token ids rather than a single value. This will require a change in ONNX Runtime GenAI to support a list of pad token ids or a change in the model builder to save only one pad token id. A similar change was made here to support a list of EOS token ids.
To avoid this error, you can set pad_token_id
in genai_config.json
to the first element in the list so that pad_token_id
is a single value instead of a list.
Thanks for the fix. I did a github pull of the latest code and created a new model. python builder.py -m ./ChatQA-1.5-8B -o chatqa -p int4 -e dml -c cache
When running the newly created model, I get a new error.
Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'NO_SUCH_FIELD: The JSON field referenced does not exist in this object.'
Using Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.2.0-rc4 in C# when running.
Exception Stack Trace:
Microsoft.ML.OnnxRuntimeGenAI.dll!Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(nint nativeResult) Line 20 C#
> Microsoft.ML.OnnxRuntimeGenAI.dll!Microsoft.ML.OnnxRuntimeGenAI.Tokenizer.Tokenizer(Microsoft.ML.OnnxRuntimeGenAI.Model model) Line 14 C#
AILocal.dll!AILocal.Onnx.OnnxLlmChatCompletion.Init() Line 25 C#
Still getting the Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'NO_SUCH_FIELD: The JSON field referenced does not exist in this object.'
with the new rc7 package.
Hey @AshD can you post your config here please?
I got the latest python builder source and recreated the model and it works now!
Uploaded the model here - https://huggingface.co/FusionQuill/Llama3-ChatQA-1.5-8B-Onnx
Model: https://huggingface.co/nvidia/ChatQA-1.5-8B Conversion Command: python -m onnxruntime_genai.models.builder -m ./ChatQA-1.5-8B -o chatqacpu -p int4 -e cpu -c cache
Runs without errors and creates the onnx model.
Ran the onnx model. C# exception:
Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'Error encountered while parsing 'C:\AI\onnx\chatqacpu\genai_config.json' JSON Error: Unknown value: pad_token_id at line 33 index 26'
Thanks, Ash