Device selection through onnxruntime-directml

[ramkrishna2910]

Hi,

I am using directml through onnxruntime and my platform has multiple directx 12 supported devices. Is there a recommended way to enumerate and select specific devices through python APIs?

Thanks

[jstoecker]

Currently, it's not possible to enumerate or select adapters through the Python API. For now, I think the only way to choose the DX device for ORT-DML in Python is by manually associating a preference for your python.exe in the graphics settings. For example:

We do want to improve this functionality in the future.

Note: This is possible in the native C/C++ API, where you can provide an adapter index or a D3D12 command queue (implicitly referencing its underlying DX adapter). The C# API also has a way to specify an adapter index, but using it effectively means writing a small native function (exposed through a DLL interface) to enumerate adapters.

[ramkrishna2910]

Thanks for the response @jstoecker Having this ability in python would be very helpful. For example, torch-directml exposes the enumeration through the following interfaces torch_directml.default_device(), torch_directml.device_count(), and torch_directml.device_name(i)

[jstoecker]

Sorry, I was mistaken. @smk2007 pointed out that we have some new APIs being added (in onnxruntime main branch, not yet released) to programmatically select an adapter based on high performance or minimum power; however, even in ORT-DML 1.16 you can select a device by index as follows:

# Specify "device_id" by index (available already in ORT 1.16)
providers = [('DmlExecutionProvider', {'device_id': '0'})]

# In a future release (ORT 1.17 possibly):
# providers = [('DmlExecutionProvider', {'performance_preference': 'high_performance'})]
# providers = [('DmlExecutionProvider', {'performance_preference': 'minimum_power'})]

session = onnxruntime.InferenceSession(model_path, options, providers=providers)

The existing "device_id" approach is not ideal, since it's not stable. For example, with a laptop, device ID mapping will likely be 0=discrete/1=integrated when the laptop is plugged in or in a high-performance mode but swapped to 0=integrated/1=discrete when the laptop is on battery. For this reason, the commented-out options above are being added.

If you have multiple discrete/high-performance GPUs, however, something like the torch_directml.device_name(...) API could be nice. For now, you can wrap some native code to enumerate adapters in the same order as ORT-DML to figure out which device ID corresponds to the device you want. I do think the specify-by-name approach is useful and something we might be able to push into an upcoming release.

[ramkrishna2910]

Yes, that would be great! And given that the device id assignments can change based on the state of the platform, the ability to enumerate the devices at runtime is critical. I am doing the following right now as a workaround:

import json
import torch_directml

if not torch_directml.is_available() or torch_directml.device_count() == 0:
    print(json.dumps({'error': 'DirectML is not available or no Directx 12 supported devices found'}))
else:
    devices = [
        {'device_id': i, 'device_name': torch_directml.device_name(i)}
        for i in range(torch_directml.device_count())
    ]
    print(json.dumps(devices))

Using the output from above to select the GPU that I want using providers = [('DmlExecutionProvider', {'device_id': '0'})] Note: Currently torch-directml only works with specific versions of torch.

[Djdefrag]

Hi everyone,

i spotted this line in onnxruntine-directml 1.17.0 changelog

Added new Python API to choose a specific GPU on multi-GPU devices with the DirectML EP.

There is some tutorial or some guide to use this new function? Browsing through the onnxruntime function I didn't find anything :(

Something like this can be really useful:

• torch_directml.device_count()
• torch_directml.device_name(index)