Support WebGPU and WebGL execution providers

[mzur]

Implement support for the onnxruntime-web WebGPU and WebGL execution providers. This will likely speed up MagicSAM a lot. WASM support should be left as a fallback.

[mzur]

There is now the webgpuworking branch that activates the WebGPU execution provider. This supposedly works with this model (not the current model we use). For the final implementation we need both, WebGPU as default but WASM as fallback (and maybe swap out the ONNX models accordingly).

[mzur]

We can also try to export the model again, keeping this comment in mind (about model checkpoint and opset compatibility): https://github.com/biigle/magic-sam/blob/107c74696b70de949ea189dc066c87541062f7f2/src/config/magic_sam.php#L45-L71

[dlangenk]

We can also try to export the model again, keeping this comment in mind (about model checkpoint and opset compatibility):

https://github.com/biigle/magic-sam/blob/107c74696b70de949ea189dc066c87541062f7f2/src/config/magic_sam.php#L45-L71

I tried that and neither on my macbook using the cpu nor in the cloud using the gpu it worked for me (the resulting onnx leads to this promise error we talked about, or is off as if the image would be smaller). I compared the opset version and they are both using v17. But the working model as linked above was produced using torch 2.0.0 and not the current version. However, I don't see a problem using the model above. If you do I could try to downgrade to pytorch 2.0.0 and see if it works then.

[mzur]

Did you just use the new ONNX or did you export it yourself with the code on Hugging Face? I'd like to have a method/script to get a working ONNX from the original model checkpoints.

[dlangenk]

The new ONNX (the huggingface one) works out of the box. I used the script included in the original segment-anything repo. And that produces a result, which does not work though. In addition, the quantization does not work out of the box anymore because of a newer onnx version which changed the quantize function parameters.

[dlangenk]

Using the notebook in the meta segment anything repo (https://github.com/facebookresearch/segment-anything/blob/main/notebooks/onnx_model_example.ipynb) on google colab it finally works. It's likely some version mismatch between cuda/torch/onnx/onnxruntime/....

Edit: They are using the exactly same versions (with the exception that the cuda version is not available on mac) PyTorch version: 2.3.1+cu121 Torchvision version: 0.18.1+cu121 Onnx: 1.16.1 Onnxruntime: 1.18.1

[dbrembilla]

I started from the webgpuworking branch and created mine to work on: webgpu-webgl-integration. However, it throws a warning that it is unable to use WebGL backend as an execution provider. I tried different browser which all should support WebGL (https://github.com/microsoft/onnxruntime/issues/20224). Looking at their issues on github, there are multiple warnings that say not to use webgl because it won't be supported for long. Considering opening an issue on their side to check whether it is a me problem.

[mzur]

If WebGL is problematic, we can ignore it for now. WebGPU will be the way forward and has priority.