microsoft / Olive

Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
https://microsoft.github.io/Olive/
MIT License
1.59k stars 166 forks source link

--use_model_builder does not work with olive auto-opt #1454

Open natke opened 3 days ago

natke commented 3 days ago

Describe the bug Crash when using the olive auto-opt --use_model_builder command

To Reproduce olive auto-opt -m meta-llama/Llama-2-7b-chat-hf --adapter_path wsvn53/Llama-2-7b-chat-lora-tricky_math -o models\Llama-2-7b-chat-LoRA --use_model_builder

Expected behavior Model is produced

Olive config No olive config

Olive logs

Loading HuggingFace model from meta-llama/Llama-2-7b-chat-hf
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\nakersha\AppData\Local\miniconda3\envs\multilora\Scripts\olive.exe\__main__.py", line 7, in <module>
  File "C:\Users\nakersha\AppData\Local\miniconda3\envs\multilora\Lib\site-packages\olive\cli\launcher.py", line 60, in main
    service.run()
  File "C:\Users\nakersha\AppData\Local\miniconda3\envs\multilora\Lib\site-packages\olive\cli\auto_opt.py", line 147, in run
    run_config = self.get_run_config(tempdir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nakersha\AppData\Local\miniconda3\envs\multilora\Lib\site-packages\olive\cli\auto_opt.py", line 174, in get_run_config
    to_replace.append(("passes", self._get_passes_config(config["passes"], olive_config)))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nakersha\AppData\Local\miniconda3\envs\multilora\Lib\site-packages\olive\cli\auto_opt.py", line 258, in _get_passes_config
    (("model_builder", "metadata_only"), self.args.input_model["type"].lower() == "onnxmodel"),
                                         ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'input_model'. Did you mean: 'input_cols'?

Other information OS: Windows Olive version: main ONNXRuntime package and version: olive-ai 0.7.0 onnx 1.17.0 onnxruntime 1.21.0.dev20241030004 onnxruntime-genai 0.5.0rc1 Transformers package version: transformers 4.44.0

Additional context Might be a dupe of #1449

shaahji commented 15 hours ago

Two additional parameters required when using DmlExecutionProvider.

I supposed the parameters should be "batch_size" and "sequence_length".

jambayk commented 6 hours ago

@shaahji DmlExecutionProvider does not require static shape. Both the model builder and the dml llm example in olive https://github.com/microsoft/Olive/blob/main/examples/directml/llm/config_llm.json don't have this. The creator of the dml model's inference session can choose to make the shapes static using sessionoptions but it is not a requirement. We need the model to have dynamic shape so that it can be used for both prompt processing and token generation.