search

deislabs / wasi-nn-onnx

Experimental ONNX implementation for WASI NN.
MIT License
46 stars 4 forks source link

wasi-nn-onnx #31

Open liborui opened 2 years ago

liborui commented 2 years ago

Hi! I am a developer/researcher on WebAssembly. I am looking for the independent runtimes of WebAssembly that could run Wasm (or wasi-nn) with GPUs. I am wondering if this repo could support? Although you said in the README that "GPU execution is not yet enabled in the native ONNX runtime (https://github.com/deislabs/wasi-nn-onnx/issues/9)", I have read the discussions on the issue and have no idea on if this project could run on the GPU?

According to my search, many runtimes (such as wasmtime) could only support the OpenVINO (Intel GPUs), I searched for the approach to run WebAssembly on other common GPUs (such as NVIDIA) or using common runtimes such as ONNX. Could you please help me with some suggestions?

radu-matei commented 2 years ago

Hi, @liborui!

As you identified, the main branch of this repository does not have CUDA enabled by default.

However, the ONNX implementation used on the host for WASI NN here can use the CUDA engine for inferencing (at least at the time of working on #9). You can use the patches in #9 as a starting point for running GPU inferences with ONNX and WASI NN, but you would most likely have to update the various ONNX runtime versions and bindings.