How to make ONNX runtime support a new GPGPU device ?

[lightnight123]

Describe the issue

we're from a new silicon company. we want to support the ONNXRUNTIME. just dont know the process for that . do you have any guide?

To reproduce

just one question.

Urgency

No response

Platform

Windows

OS Version

11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

10

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Other / Unknown

Execution Provider Library Version

No response

[skottmckay]

Typically support for specific devices is done by creating an execution provider.

https://onnxruntime.ai/docs/execution-providers/ https://onnxruntime.ai/docs/execution-providers/add-execution-provider.html

The GetCapability method implements the logic to tell ORT which nodes you can execute (typically based on the ONNX operator and opset, and the input/output types). If the implementation uses a static kernel those are registered when the execution provider is created. You can also create a compiled node from one or more onnx nodes (the values returned by GetCapability will determine if IExecutionProvider::Compile is called). Also fine to have a mix of static and compiled handling.

There's a very simplistic stub implementation of an EP used in testing that might be a good place to start to get an idea of the pieces to implement. It can optionally do things multiple ways (static kernels, compiled kernels) so will cover a few different potential approaches.

https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/providers/internal_testing

After you're familiar with that you can take a look at how other EPs are implemented. CPU EP uses static kernels only. Something like NNAPI or CoreML creates a compiled kernel for the NNAPI/CoreML model that is generated from multiple nodes in the original model.

@pranavsharma should new EPs use the provider bridge so they can be separated from the core code?

[pranavsharma]

We're working on an EP interface that won't require building an EP with the core code. This means you can develop and build an EP independently in a separate repo and have ORT link with it. The interface won't be ABI. If you're not hard pressed for time I would advise you to wait for that. cc @souptc For now, using the provider bridge would be a good start so that you can have 2 separate DLLs - ORT and your_ep.dll. This allows flexibility in packaging.

[github-actions[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.