search

microsoft / onnxruntime-genai

Generative AI extensions for onnxruntime
MIT License
511 stars 127 forks source link

Device API throws exception #488

Open natke opened 5 months ago

natke commented 5 months ago

From the last question in #430

natke commented 5 months ago

From @ashD

@yufenglee I tried calling Utils.GetCurrentGpuDeviceId() in C# using the Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.2.0-rc6 nuget

and it threw a Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'CUDA and/or ROCM execution provider is either not enabled or not available.'

Same error on Utils.SetCurrentGpuDeviceId(1)

baijumeswani commented 5 months ago

@natke is this still an issue?

AshD commented 5 months ago

Just ran the same code with DirectML nuget 0.3.0-rc2. Same error.

natke commented 4 months ago

Reproduced with 0.3.0

skyline75489 commented 4 months ago

This is somewhat expected with DML package. The implementation in ORT only supports CUDA and ROCm:

https://github.com/microsoft/onnxruntime/blob/4ac4cd26687ce1b15c22206f9e281443639aa5e0/onnxruntime/core/session/provider_bridge_ort.cc#L2177

If we want to add DML support, we need to add it inside ORT itself.

See also: #566

typicaldigital commented 4 months ago

Dear team, any update when this bug will be fixed? This prevents anyone with multiple GPU's from selecting the best one for inference.

typicaldigital commented 3 months ago

@baijumeswani Any update on this issue?

baijumeswani commented 2 months ago

@PatriceVignola is it possible to set the gpu device id for dml? Can we expose it through onnxruntime-genai?