jambayk
commented
3 months ago Hi,
could you try the workflow again by adding "use_gpu": false under insert_beam_search.config? The whisper example has not been tested with DML ep so the insert beam search assumes it's only run with cpu or cuda ep
this part adds DecoderMaskedMultiHeadAttention to the model but the operator doesn't appear to be implemented for DML ep https://github.com/microsoft/Olive/blob/7fa2c4138f085c339b9c5da5d4a57fadd2de104a/olive/passes/onnx/insert_beam_search.py#L272
WA225
xiaoyu-work
heibaidaolx123
Describe the bug I am trying to run Whisper on an AMD Radeon 780M Graphics using DirectML EP but it is showing the Not Implemented error below.
To Reproduce python -m pip install onnxruntime-directml python -m olive.workflows.run --config whisper_dml_fp32.json --setup python -m pip install onnxruntime_extensions python -m olive.workflows.run --config whisper_dml_fp32.json
Olive config whisper_dml_fp32.json: { "input_model": { "type": "PyTorchModel", "config": { "model_script": "code/user_script.py", "script_dir": "code", "hf_config": { "model_class": "WhisperForConditionalGeneration", "model_name": "openai/whisper-tiny.en", "components": [ { "name": "encoder_decoder_init", "io_config": "get_encdec_io_config", "component_func": "get_encoder_decoder_init", "dummy_inputs_func": "encoder_decoder_init_dummy_inputs" }, { "name": "decoder", "io_config": "get_dec_io_config", "component_func": "get_decoder", "dummy_inputs_func": "decoder_dummy_inputs" } ], "from_pretrained_args": { "attn_implementation": "eager" } } } }, "systems": { "local_system": { "type": "LocalSystem", "config": { "accelerators": [ { "device": "gpu", "execution_providers": [ "DmlExecutionProvider" ] } ] } } }, "evaluators": { "common_evaluator": { "metrics": [ { "name": "latency", "type": "latency", "sub_types": [ { "name": "avg", "priority": 1 } ], "user_config": { "user_script": "code/user_script.py", "script_dir": "code", "data_dir": "data", "dataloader_func": "whisper_dataloader", "func_kwargs": { "dataloader_func": { "model_name": "openai/whisper-tiny.en", "use_audio_decoder": true } } } } ] } }, "passes": { "conversion": { "type": "OnnxConversion", "config": { "target_opset": 17 } }, "transformers_optimization": { "type": "OrtTransformersOptimization", "config": { "optimization_options": { "use_multi_head_attention": true }, "use_gpu": true } }, "insert_beam_search": { "type": "InsertBeamSearch", "config": { "use_forced_decoder_ids": false, "use_logits_processor": false, "fp16": false } }, "prepost": { "type": "AppendPrePostProcessingOps", "config": { "tool_command": "whisper", "tool_command_args": { "model_name": "openai/whisper-tiny.en", "use_audio_decoder": true }, "target_opset": 17 } } }, "engine": { "log_severity_level": 0, "host": "local_system", "target": "local_system", "evaluator": "common_evaluator", "evaluate_input_model": false, "clean_cache": false, "cache_dir": "cache", "output_dir": "models", "output_name": "whisper_dml_fp32" } }
Olive logs After it reached the step in which it tries to run Olive on gpu-dml, it fails giving this error: onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for DecoderMaskedMultiHeadAttention(1) node with name 'Attention_0'
Other information