search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.06k stars 2.83k forks source link

[Build] QNN EP issue #20206

Open tomas-pet opened 5 months ago

tomas-pet commented 5 months ago

Describe the issue

Getting this error when I run my python app: python .\roberta.py 2024-04-04 21:24:37.4152414 [E:onnxruntime:, qnn_execution_provider.cc:499 onnxruntime::QNNExecutionProvider::GetCapability] QNN SetupBackend failed qnn_backend_manager.cc:56 onnxruntime::qnn::QnnBackendManager::GetQnnInterfaceProvider Unable to load backend, error: load library failed

This is what I added in the onnxruntime inference session line of code: provider="QNNExecutionProvider", provider_options={"backend_path": "\Qualcomm\AIStack\QNN\2.19.2.240210\lib\aarch64-windows-msvc\QnnHtp.dll"}

I did the following to install QNN EP:

1) Installed QNN 2.19.2 from QPM 2) Copied v73skel.so file to my aarch64-windows-msvc folder 3) Added aarch64-windows-msvc directory to path 4) python -m pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ ort-nightly-qnn

Urgency

This is very urgent. We are looking to make a release soon.

Target platform

SC8380XP - Snapdragon X Elite

Build script

I did the following to install QNN EP:

1) Installed QNN 2.19.2 from QPM 2) Copied v73skel.so file to my aarch64-windows-msvc folder 3) Added aarch64-windows-msvc directory to path 4) python -m pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ ort-nightly-qnn

Error / output

2024-04-04 21:24:37.4152414 [E:onnxruntime:, qnn_execution_provider.cc:499 onnxruntime::QNNExecutionProvider::GetCapability] QNN SetupBackend failed qnn_backend_manager.cc:56 onnxruntime::qnn::QnnBackendManager::GetQnnInterfaceProvider Unable to load backend, error: load library failed.

Visual Studio Version

No response

GCC / Compiler Version

No response

HectorSVC commented 5 months ago

please also copy hexagon-v73/unsigned/libqnnhtpv73.cat

HectorSVC commented 5 months ago

I need to double check the QNN version used for QNN python package nightly build. Please try qnn-v2.18.0.240101 if it still not work.

HectorSVC commented 5 months ago

I would recommend you build from source using our main branch. There are some new features we committed recently.

tomas-pet commented 5 months ago

I tried with QNN-2.18 I still get same error. I don't want to really build from source. I would prefer to use pip install. Can you please provide me with validated steps? This is urgent.

HectorSVC commented 5 months ago

Please run pip show ort-nightly-qnn to get the exact version. Is it 1.18.0.dev20240403004?

tomas-pet commented 5 months ago

Yes it is this exact version. On Apr 5, 2024, at 3:41 PM, Hector Li @.***> wrote: Please run pip show ort-nightly-qnn to get the exact version. Is it 1.18.0.dev20240403004?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

HectorSVC commented 5 months ago

You need to start a new cmd after changing the environment path. I can run with it without any problem following your steps with qnn-v2.18.0.240101.

tomas-pet commented 5 months ago

Can you share how your file path looks like in backend_path?

HectorSVC commented 5 months ago

sess = rt.InferenceSession('inception-v1-12.quant.onnx', so, providers=['QNNExecutionProvider'], provider_options=[{'backend_path' : "QnnHtp.dll"}]) This the python code since the aarch64-windows-msvc folder is in environment path already.

tomas-pet commented 5 months ago

Thank you! Not sure why this is still not working for me.

tomas-pet commented 5 months ago

Is there an easy way to find out HTP arch value?

HectorSVC commented 5 months ago

https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/overview.html#supported-snapdragon-devices

tomas-pet commented 5 months ago

Thanks, so ours is v73. No idea why this is not working. I'll give up now. Thanks for your help!

HectorSVC commented 5 months ago

suggest you run some QNN tutorials first to make sure everything is OK. https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/tutorials.html

tomas-pet commented 5 months ago

Can you provide full list of instructions from start to finish for installing and setting up QNN EP using pip? I am not only one struggling, my entire team is not able to bring this up.

Also, please let me know how to uninstall current QNN version. I will start from scratch using your steps.

HectorSVC commented 5 months ago

could you share the output of command pip show ort-nightly-qnn.

tomas-pet commented 5 months ago

Name: ort-nightly-qnn Version: 1.18.0.dev20240403004 Summary: ONNX Runtime is a runtime accelerator for Machine Learning models Home-page: https://onnxruntime.ai Author: Microsoft Corporation Author-email: onnxruntime@microsoft.com License: MIT License Location: c:\users\hcktest\appdata\local\programs\python\python38\lib\site-packages Requires: flatbuffers, sympy, numpy, packaging, coloredlogs, protobuf Required-by:

HectorSVC commented 5 months ago

I see. You python is x64? Your device is Arm64 right? you need to install python-arm64, please use 3.11

HectorSVC commented 5 months ago

Let's see if we can get more information by enabling the verbose log:

import onnxruntime as rt

so = rt.SessionOptions() so.log_severity_level = 0 sess = rt.InferenceSession('inception-v1-12.quant.onnx', so, providers=['QNNExecutionProvider'], provider_options=[{'backend_path' : "QnnHtp.dll"}])

tomas-pet commented 5 months ago

I am getting this issue now: C:\Users\HCKTest\Downloads>py -m pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ ort-nightly-qnn Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ Collecting ort-nightly-qnn Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240403004/ort_nightly_qnn-1.18.0.dev20240403004-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 22.4 MB/s eta 0:00:00 INFO: pip is looking at multiple versions of ort-nightly-qnn to determine which version is compatible with other requirements. This could take a while. Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240402005/ort_nightly_qnn-1.18.0.dev20240402005-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 23.6 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240329005/ort_nightly_qnn-1.18.0.dev20240329005-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 23.5 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240329004/ort_nightly_qnn-1.18.0.dev20240329004-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 13.5 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240329001/ort_nightly_qnn-1.18.0.dev20240329001-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 24.8 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240328001/ort_nightly_qnn-1.18.0.dev20240328001-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 26.3 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240316001/ort_nightly_qnn-1.18.0.dev20240316001-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 17.1 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240314003/ort_nightly_qnn-1.18.0.dev20240314003-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 24.8 MB/s eta 0:00:00 INFO: pip is still looking at multiple versions of ort-nightly-qnn to determine which version is compatible with other requirements. This could take a while. Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240313005/ort_nightly_qnn-1.18.0.dev20240313005-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 13.1 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240312004/ort_nightly_qnn-1.18.0.dev20240312004-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 22.2 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240311001/ort_nightly_qnn-1.18.0.dev20240311001-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 14.3 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240307004/ort_nightly_qnn-1.18.0.dev20240307004-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 5.4 MB/s eta 0:00:00 Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240306006/ort_nightly_qnn-1.18.0.dev20240306006-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 12.4 MB/s eta 0:00:00 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C. Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/ort-nightly-qnn/1.18.dev20240305002/ort_nightly_qnn-1.18.0.dev20240305002-cp311-cp311-win_arm64.whl (7.0 MB) ---------------------------------------- 7.0/7.0 MB 13.9 MB/s eta 0:00:00

tomas-pet commented 5 months ago

ERROR: Cannot install ort-nightly-qnn==1.18.0.dev20240125005, ort-nightly-qnn==1.18.0.dev20240126005, ort-nightly-qnn==1.18.0.dev20240129005, ort-nightly-qnn==1.18.0.dev20240130006, ort-nightly-qnn==1.18.0.dev20240201001, ort-nightly-qnn==1.18.0.dev20240201005, ort-nightly-qnn==1.18.0.dev20240202004, ort-nightly-qnn==1.18.0.dev20240205005, ort-nightly-qnn==1.18.0.dev20240206005, ort-nightly-qnn==1.18.0.dev20240207003, ort-nightly-qnn==1.18.0.dev20240209007, ort-nightly-qnn==1.18.0.dev20240210001, ort-nightly-qnn==1.18.0.dev20240213001, ort-nightly-qnn==1.18.0.dev20240213003, ort-nightly-qnn==1.18.0.dev20240214003, ort-nightly-qnn==1.18.0.dev20240215004, ort-nightly-qnn==1.18.0.dev20240216004, ort-nightly-qnn==1.18.0.dev20240217001, ort-nightly-qnn==1.18.0.dev20240218001, ort-nightly-qnn==1.18.0.dev20240219001, ort-nightly-qnn==1.18.0.dev20240221006, ort-nightly-qnn==1.18.0.dev20240223001, ort-nightly-qnn==1.18.0.dev20240223005, ort-nightly-qnn==1.18.0.dev20240224001, ort-nightly-qnn==1.18.0.dev20240225001, ort-nightly-qnn==1.18.0.dev20240226004, ort-nightly-qnn==1.18.0.dev20240227006, ort-nightly-qnn==1.18.0.dev20240228007, ort-nightly-qnn==1.18.0.dev20240229007, ort-nightly-qnn==1.18.0.dev20240301005, ort-nightly-qnn==1.18.0.dev20240304002, ort-nightly-qnn==1.18.0.dev20240305002, ort-nightly-qnn==1.18.0.dev20240306006, ort-nightly-qnn==1.18.0.dev20240307004, ort-nightly-qnn==1.18.0.dev20240311001, ort-nightly-qnn==1.18.0.dev20240312004, ort-nightly-qnn==1.18.0.dev20240313005, ort-nightly-qnn==1.18.0.dev20240314003, ort-nightly-qnn==1.18.0.dev20240316001, ort-nightly-qnn==1.18.0.dev20240328001, ort-nightly-qnn==1.18.0.dev20240329001, ort-nightly-qnn==1.18.0.dev20240329004, ort-nightly-qnn==1.18.0.dev20240329005, ort-nightly-qnn==1.18.0.dev20240402005 and ort-nightly-qnn==1.18.0.dev20240403004 because these package versions have conflicting dependencies.

The conflict is caused by: ort-nightly-qnn 1.18.0.dev20240403004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240402005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240329005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240329004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240329001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240328001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240316001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240314003 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240313005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240312004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240311001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240307004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240306006 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240305002 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240304002 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240301005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240229007 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240228007 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240227006 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240226004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240225001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240224001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240223005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240223001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240221006 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240219001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240218001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240217001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240216004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240215004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240214003 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240213003 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240213001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240210001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240209007 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240207003 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240206005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240205005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240202004 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240201005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240201001 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240130006 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240129005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240126005 depends on coloredlogs ort-nightly-qnn 1.18.0.dev20240125005 depends on coloredlogs

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

HectorSVC commented 5 months ago

install these first: numpy coloredlogs flatbuffers packaging protobuf sympy

HectorSVC commented 5 months ago

Were you in wrong python environment previously?

tomas-pet commented 5 months ago

my python environment is fine - still getting same error. here is a bit from verbose logs:

2024-04-05 18:16:13.8561453 [I:onnxruntime:, inference_session.cc:532 onnxruntime::InferenceSession::TraceSessionOptions] Session Options { execution_mode:0 execution_order:DEFAULT enable_profiling:0 optimized_model_filepath: enable_mem_pattern:1 enable_mem_reuse:1 enable_cpu_mem_arena:1 profile_file_prefix:onnxruntimeprofile session_logid: session_log_severity_level:0 session_log_verbosity_level:0 max_num_graph_transformation_steps:10 graph_optimization_level:3 intra_op_param:OrtThreadPoolParams { thread_pool_size: 0 auto_set_affinity: 0 allow_spinning: 1 dynamic_blockbase: 0 stack_size: 0 affinity_str: set_denormal_as_zero: 0 } inter_op_param:OrtThreadPoolParams { thread_pool_size: 0 auto_set_affinity: 0 allow_spinning: 1 dynamic_blockbase: 0 stack_size: 0 affinity_str: set_denormal_as_zero: 0 } use_per_session_threads:1 thread_pool_allow_spinning:1 use_deterministic_compute:0 config_options: { } } 2024-04-05 18:16:13.9023104 [I:onnxruntime:, inference_session.cc:432 onnxruntime::InferenceSession::ConstructorCommon::::operator ()] Flush-to-zero and denormal-as-zero are off 2024-04-05 18:16:13.9087983 [I:onnxruntime:, inference_session.cc:440 onnxruntime::InferenceSession::ConstructorCommon] Creating and using per session threadpools since use_per_sessionthreads is true 2024-04-05 18:16:13.9125045 [I:onnxruntime:, inference_session.cc:458 onnxruntime::InferenceSession::ConstructorCommon] Dynamic block base set to 0 2024-04-05 18:16:14.1864266 [I:onnxruntime:, inference_session.cc:1601 onnxruntime::InferenceSession::Initialize] Initializing session. 2024-04-05 18:16:14.1892573 [I:onnxruntime:, inference_session.cc:1638 onnxruntime::InferenceSession::Initialize] Adding default CPU execution provider. 2024-04-05 18:16:14.2004814 [I:onnxruntime:, graph_partitioner.cc:900 onnxruntime::GraphPartitioner::InlineFunctionsAOT] This model does not have any local functions defined. AOT Inlining is not performed 2024-04-05 18:16:14.2081481 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.2134186 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 1 with status: OK 2024-04-05 18:16:14.2269641 [I:onnxruntime:, graph.cc:3597 onnxruntime::Graph::CleanUnusedInitializersAndNodeArgs] Removing initializer '248'. It is no longer used by any node. 2024-04-05 18:16:14.2310337 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.2392564 [I:onnxruntime:, constant_sharing.cc:256 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 375 2024-04-05 18:16:14.2435108 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.2559162 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.2621270 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 1 with status: OK 2024-04-05 18:16:14.2798411 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 1 with status: OK 2024-04-05 18:16:14.2912403 [I:onnxruntime:, graph.cc:3597 onnxruntime::Graph::CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_7_0_1_2'. It is no longer used by any node. 2024-04-05 18:16:14.2965752 [I:onnxruntime:, graph.cc:3597 onnxruntime::Graph::CleanUnusedInitializersAndNodeArgs] Removing initializer '226'. It is no longer used by any node. 2024-04-05 18:16:14.3015632 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.3064004 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 293 2024-04-05 18:16:14.3093840 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 333 2024-04-05 18:16:14.3132871 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 314 2024-04-05 18:16:14.3164333 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 357 2024-04-05 18:16:14.3195439 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 422 2024-04-05 18:16:14.3229076 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 462 2024-04-05 18:16:14.3247816 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 443 2024-04-05 18:16:14.3266979 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 486 2024-04-05 18:16:14.3285938 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 551 2024-04-05 18:16:14.3302163 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 591 2024-04-05 18:16:14.3354708 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 572 2024-04-05 18:16:14.3373462 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 615 2024-04-05 18:16:14.3396490 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 680 2024-04-05 18:16:14.3417860 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 720 2024-04-05 18:16:14.3450326 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 701 2024-04-05 18:16:14.3465687 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 744 2024-04-05 18:16:14.3484532 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 809 2024-04-05 18:16:14.3500582 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 849 2024-04-05 18:16:14.3517319 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 830 2024-04-05 18:16:14.3554513 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 873 2024-04-05 18:16:14.3583047 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 938 2024-04-05 18:16:14.3614866 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 978 2024-04-05 18:16:14.3654807 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 959 2024-04-05 18:16:14.3685370 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1002 2024-04-05 18:16:14.3706969 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1067 2024-04-05 18:16:14.3748085 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1107 2024-04-05 18:16:14.3779280 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1088 2024-04-05 18:16:14.3798648 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1131 2024-04-05 18:16:14.3842586 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1196 2024-04-05 18:16:14.3867105 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1236 2024-04-05 18:16:14.3896369 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1217 2024-04-05 18:16:14.3938190 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1260 2024-04-05 18:16:14.3958125 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1325 2024-04-05 18:16:14.3991188 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1365 2024-04-05 18:16:14.4032303 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1346 2024-04-05 18:16:14.4057237 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1389 2024-04-05 18:16:14.4086169 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1454 2024-04-05 18:16:14.4128576 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1494 2024-04-05 18:16:14.4157956 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1475 2024-04-05 18:16:14.4186185 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1518 2024-04-05 18:16:14.4207608 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1583 2024-04-05 18:16:14.4255424 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1623 2024-04-05 18:16:14.4283320 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1604 2024-04-05 18:16:14.4318921 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1647 2024-04-05 18:16:14.4350312 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1712 2024-04-05 18:16:14.4370277 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1752 2024-04-05 18:16:14.4413863 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1733 2024-04-05 18:16:14.4441776 [I:onnxruntime:, reshape_fusion.cc:49 onnxruntime::ReshapeFusion::ApplyImpl] Fused reshape node: 1776 2024-04-05 18:16:14.4458818 [I:onnxruntime:, reshape_fusion.cc:55 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 48 2024-04-05 18:16:14.4506564 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 1 with status: OK 2024-04-05 18:16:14.4602647 [I:onnxruntime:, graph.cc:3597 onnxruntime::Graph::CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_7_1_1_3'. It is no longer used by any node. 2024-04-05 18:16:14.4641547 [I:onnxruntime:, graph.cc:3597 onnxruntime::Graph::CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_7_1_1_1'. It is no longer used by any node. 2024-04-05 18:16:14.4688360 [I:onnxruntime:, graph.cc:3597 onnxruntime::Graph::CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_7_1_1_2'. It is no longer used by any node. 2024-04-05 18:16:14.4749711 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.4810676 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.4857151 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.4908129 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.4960432 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer TransposeOptimizer modified: 0 with status: OK 2024-04-05 18:16:14.5021119 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.5057511 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.5121686 [I:onnxruntime:, constant_sharing.cc:256 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 47 2024-04-05 18:16:14.5157174 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.5265064 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.5324610 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.5368061 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.5419906 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.5455346 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.5502273 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.5540390 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.5598994 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.5632644 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.5690339 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.5723046 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.5783887 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.5892927 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.5942507 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.5996073 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.6036500 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.6087332 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.6119274 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.6178487 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.6224548 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.6275762 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.6306473 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.6359637 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.6396970 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.6502906 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.6549051 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.6597612 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.6644157 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.6686121 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.6711461 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.6769586 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.6805021 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.6858398 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.6889121 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.6906675 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.6946018 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.7030598 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.7083175 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.7132791 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.7178755 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.7227460 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.7266135 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.7300885 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.7359887 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.7398558 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.7455539 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.7512622 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.7549978 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.7659662 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.7711770 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.7754562 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.7820650 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.7870730 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.7926310 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.7967404 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.8025657 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.8057444 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.8115960 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.8155334 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.8210378 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.8318608 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.8373768 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.8422963 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.8458040 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.8515988 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.8545118 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.8603391 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.8638331 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.8691495 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.8721962 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.8779757 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.8821367 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.8934275 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.8995626 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.9037805 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.9096449 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.9133051 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.9169112 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.9215821 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.9270916 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.9304130 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.9360501 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:14.9396858 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:14.9455292 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:14.9568021 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:14.9617964 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:14.9673705 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:14.9707883 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:14.9762856 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:14.9793258 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:14.9850224 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:14.9889378 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:14.9941538 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:14.9975055 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK 2024-04-05 18:16:15.0030495 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK 2024-04-05 18:16:15.0075543 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantSharing modified: 1 with status: OK 2024-04-05 18:16:15.0183550 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ShapeInputMerge modified: 0 with status: OK 2024-04-05 18:16:15.0240604 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK 2024-04-05 18:16:15.0284851 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ConstantFolding modified: 0 with status: OK 2024-04-05 18:16:15.0330188 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer MatMulAddFusion modified: 0 with status: OK 2024-04-05 18:16:15.0368096 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer ReshapeFusion modified: 0 with status: OK 2024-04-05 18:16:15.0414228 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK 2024-04-05 18:16:15.0456341 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer QDQPropagationTransformer modified: 0 with status: OK 2024-04-05 18:16:15.0510312 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK 2024-04-05 18:16:15.0546237 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer RocmBlasAltImpl modified: 0 with status: OK 2024-04-05 18:16:15.0612537 [E:onnxruntime:, qnn_execution_provider.cc:499 onnxruntime::QNNExecutionProvider::GetCapability] QNN SetupBackend failed qnn_backend_manager.cc:56 onnxruntime::qnn::QnnBackendManager::GetQnnInterfaceProvider Unable to load backend, error: load library failed 2024-04-05 18:16:15.0760918 [I:onnxruntime:, graph_transformer.cc:15 onnxruntime::GraphTransformer::Apply] GraphTransformer TransposeOptimizer_CPUExecutionProvider modified: 0 with status: OK

HectorSVC commented 5 months ago

Seems mostly likely relate to driver or QNN setup. What's your OS version? Have you ever successfully run any model with SNPE or QNN?

tomas-pet commented 5 months ago

Yes, I have run with QNN. OS is Windows 11

tomas-pet commented 5 months ago

I got it to work finally but now I am getting: Traceback (most recent call last): File "roberta.py", line 8, in model = ORTModelForQuestionAnswering.from_pretrained("optimum/roberta-base-squad2", provider="QNNExecutionProvider", provider_options={'backend_path': "QnnCpu.dll"}) File "C:\Users\HCKTest\AppData\Local\Programs\Python\Python38\lib\site-packages\optimum\onnxruntime\modeling_ort.py", line 663, in from_pretrained return super().from_pretrained( File "C:\Users\HCKTest\AppData\Local\Programs\Python\Python38\lib\site-packages\optimum\modeling_base.py", line 401, in from_pretrained return from_pretrained_method( File "C:\Users\HCKTest\AppData\Local\Programs\Python\Python38\lib\site-packages\optimum\onnxruntime\modeling_ort.py", line 511, in _from_pretrained model = ORTModel.load_model( File "C:\Users\HCKTest\AppData\Local\Programs\Python\Python38\lib\site-packages\optimum\onnxruntime\modeling_ort.py", line 374, in load_model return ort.InferenceSession( File "C:\Users\HCKTest\AppData\Local\Programs\Python\Python38\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "C:\Users\HCKTest\AppData\Local\Programs\Python\Python38\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : qnn_model.cc:71 onnxruntime::qnn::QnnModel::ParseGraphInputOrOutput Dynamic shape is not supported yet, for output: 243

HectorSVC commented 5 months ago

What did you do? So, QNN CPU backend worked. How about HTP backend. QNN doesn't support dynamic shape.

HectorSVC commented 5 months ago

Hi, @tomas-pet, want to check with you how is it going? We may need to involve Qualcomm guys if you still see issue with HTP backend.

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.