Created onnx version of ChatQA-1.5-8B model with Unknown value: pad_token_id error on inferencing

[AshD]

Model: https://huggingface.co/nvidia/ChatQA-1.5-8B Conversion Command: python -m onnxruntime_genai.models.builder -m ./ChatQA-1.5-8B -o chatqacpu -p int4 -e cpu -c cache

Runs without errors and creates the onnx model.

Ran the onnx model. C# exception: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'Error encountered while parsing 'C:\AI\onnx\chatqacpu\genai_config.json' JSON Error: Unknown value: pad_token_id at line 33 index 26'

Thanks, Ash

[natke]

Hi @AshD We are looking into this

[baijumeswani]

@AshD what version of onnxruntime-genai are you using?

[AshD]

Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.2.0-rc4 in C# when running.

Latest onnxruntime-genai installed from pip for the python onnxruntime_genai.models.builder command

[kunal-vaishnavi]

The genai_config.json file shows a list of pad token ids rather than a single value. This will require a change in ONNX Runtime GenAI to support a list of pad token ids or a change in the model builder to save only one pad token id. A similar change was made here to support a list of EOS token ids.

To avoid this error, you can set pad_token_id in genai_config.json to the first element in the list so that pad_token_id is a single value instead of a list.

[AshD]

Thanks for the fix. I did a github pull of the latest code and created a new model. python builder.py -m ./ChatQA-1.5-8B -o chatqa -p int4 -e dml -c cache

When running the newly created model, I get a new error.

Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'NO_SUCH_FIELD: The JSON field referenced does not exist in this object.'

Using Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.2.0-rc4 in C# when running.

[AshD]

Exception Stack Trace:

    Microsoft.ML.OnnxRuntimeGenAI.dll!Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(nint nativeResult) Line 20 C#
>   Microsoft.ML.OnnxRuntimeGenAI.dll!Microsoft.ML.OnnxRuntimeGenAI.Tokenizer.Tokenizer(Microsoft.ML.OnnxRuntimeGenAI.Model model) Line 14  C#
    AILocal.dll!AILocal.Onnx.OnnxLlmChatCompletion.Init() Line 25   C#

[AshD]

Still getting the Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'NO_SUCH_FIELD: The JSON field referenced does not exist in this object.'

with the new rc7 package.

[natke]

Hey @AshD can you post your config here please?

[AshD]

I got the latest python builder source and recreated the model and it works now!

Uploaded the model here - https://huggingface.co/FusionQuill/Llama3-ChatQA-1.5-8B-Onnx