Open natke opened 3 days ago
Two additional parameters required when using DmlExecutionProvider.
I supposed the parameters should be "batch_size" and "sequence_length".
@shaahji DmlExecutionProvider does not require static shape. Both the model builder and the dml llm example in olive https://github.com/microsoft/Olive/blob/main/examples/directml/llm/config_llm.json don't have this. The creator of the dml model's inference session can choose to make the shapes static using sessionoptions but it is not a requirement. We need the model to have dynamic shape so that it can be used for both prompt processing and token generation.
Describe the bug Crash when using the olive auto-opt --use_model_builder command
To Reproduce
olive auto-opt -m meta-llama/Llama-2-7b-chat-hf --adapter_path wsvn53/Llama-2-7b-chat-lora-tricky_math -o models\Llama-2-7b-chat-LoRA --use_model_builder
Expected behavior Model is produced
Olive config No olive config
Olive logs
Other information OS: Windows Olive version: main ONNXRuntime package and version: olive-ai 0.7.0 onnx 1.17.0 onnxruntime 1.21.0.dev20241030004 onnxruntime-genai 0.5.0rc1 Transformers package version: transformers 4.44.0
Additional context Might be a dupe of #1449