search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.65k stars 2.93k forks source link

How to determine which provider current inference is using? #22243

Open SolveProb opened 1 month ago

SolveProb commented 1 month ago

Hi, I am a newcomer to the community and have some questions to ask.

  1. How to determine which provider the current inference is running on?
  2. How to switch from CPU to GPU provider?
  3. How to set the priority of different providers?

The phenomenon I encountered is that for some reason, the inference process automatically switched from GPU to CPU. The specific phenomenon is: From 0 to 60 minutes, GPU memory occupied 500M, GPU utilization was about 50%, and CPU utilization was about 30% After 60 minutes, GPU memory occupied 500M, GPU utilization was dropped to 0%, and CPU utilization was increased to 100% Therefore, now I want to confirm which provider is used for each inference, monitor each inference, and automatically switch it to GPU provider if the last inference is on CPU.

After some searching, I found this GetAvailableProviders() function in the document, but the result it returns does not seem to accurately reflect the providers that can currently be executed normally.

I checked this issue 486 and the document, but still couldn't find the desired interface.

Currently using onnx 1.12.1 version using C++ API

Thanks very much.

henryruhs commented 1 month ago

Once you created a onnxruntime.InferenceSession there is the method get_providers that can be used.

Documentation: https://onnxruntime.ai/docs/api/python/api_summary.html#inferencesession

SolveProb commented 1 month ago

Once you created a onnxruntime.InferenceSession there is the method get_providers that can be used.

Documentation: https://onnxruntime.ai/docs/api/python/api_summary.html#inferencesession

Thank you for your kind reminder. I am using the C++ API and did not find the relevant function in the C++ documentation.

github-actions[bot] commented 3 days ago

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.