search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
13.92k stars 2.81k forks source link

DML incorrect results, probably Split node #21007

Open jlami opened 2 months ago

jlami commented 2 months ago

Describe the issue

Running the htdemucs model on DML gives incorrect results compared to the CPU execution.

To reproduce

The model we are using is the htdemucs v4 model from here: https://github.com/facebookresearch/demucs

You can find a converted onnx model in this zip: https://github.com/intel/openvino-plugins-ai-audacity/releases/download/v3.4.2-R1/openvino-models.zip (2GB zip, htdemucs_v4.onnx itself is 170MB)

If I comment out this line, DML gives correct results: https://github.com/microsoft/onnxruntime/blob/74028e4bdcfa3069113b657bc1e2f32389372dfa/onnxruntime/core/providers/dml/DmlExecutionProvider/src/Operators/OperatorRegistration.cpp#L782

Urgency

No response

Platform

Windows

OS Version

10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

74028e4bdcfa3069113b657bc1e2f32389372dfa

ONNX Runtime API

C

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

1.14

jlami commented 2 months ago

Setting ep.dml.disable_graph_fusion to 1 while enabling the Split op again also gives correct results, so I guess the problem is in the DmlGraphFusionTransformer

sophies927 commented 2 months ago

@smk2007

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

jlami commented 1 week ago

Is there anything I can do to help with this issue? Or do you have any pointers as to where I could look myself. I could not really make any sense of the graph fusion yet, so any pointers are welcome.