search

xenova / transformers.js

State-of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server!
https://huggingface.co/docs/transformers.js
Apache License 2.0
11.32k stars 703 forks source link

Does `WEBGPU` Truly Enhance Inference Time Acceleration? #586

Open kishorekaruppusamy opened 8 months ago

kishorekaruppusamy commented 8 months ago

Question

Recently, I've been extensively utilizing transformers.js to load transformer models, and Kudos to the team for this wonderful library ... Specifically, I've been experimenting with version 2.15.0 of transformers.js.

Despite the fact that the model runs on the web-assembly backend, I've noticed some slowness in inference. In an attempt to address this issue, I experimented withwebgpu inference using the v3 branch. However, the inference time did not meet my expectations.

Is it possible for webgpu to significantly accelerate the inference time?

kishorekaruppusamy commented 8 months ago

Used model : Xenova/Whisper-tiny.en , Xenova/all-MiniLM-L6-v2 Model_quantization : quantized_model transformer-js version : 3.0.0-alpha.0 executionProviders : ['webgpu']; Hardware : MacBook M1 Pro with 10-core CPU and 16-core GPU Ram : 16GB

is there any way to accelerate the speed of inference ??

xenova commented 7 months ago

Encoder-decoder models are still a work in progress, but the bert-based embedding models work very well! For example, I get >100x improvement with all-MiniLM-L6-v2.

image

Try it out yourself: https://huggingface.co/spaces/Xenova/webgpu-embedding-benchmark

wujohns commented 7 months ago

Whether nodejs can also benefit from this speedup

felladrin commented 7 months ago

Whether nodejs can also benefit from this speedup

I'm not sure if Node.js can benefit from this speedup, but it is possible that Deno can.

hans00 commented 7 months ago

Whether nodejs can also benefit from this speedup

onnxruntime-node not support WebGPU, but it support DirectML (Windows) or CUDA (Linux) (Official prebuilt)

wujohns commented 7 months ago

Whether nodejs can also benefit from this speedup

onnxruntime-node not support WebGPU, but it support DirectML (Windows) or CUDA (Linux)

But there is no device setting (ex: cuda) for transformerjs

jhpassion0621 commented 7 months ago

@xenova When can I test encoder-decoder model with WebGPU? I can't wait anymore. I am very excited to see that asap.

talavivi03 commented 6 months ago

Just tried it out, and wow, it's a huge upgrade! When are you thinking of launching it?