Is possible port it to WASM?

OpenNMT / CTranslate2

Fast inference engine for Transformer models

https://opennmt.net/CTranslate2

MIT License

3.35k stars 293 forks source link

Is possible port it to WASM? #528

Open vitonsky opened 3 years ago

vitonsky commented 3 years ago

Hi, in other issue to me say that probably i should will compile CTranslate2 to WASM

Source https://github.com/LibreTranslate/LibreTranslate/issues/120#issuecomment-888676870

I here because at this time i don't know how work CTranslate2 and see in README that it use GPU, then i have question.

Is possible use CTranslate2 for WASM or it will work not performant or even not work due to GPU bound?

What you think about it?

guillaumekln commented 3 years ago

CTranslate2 does not require a GPU. It can also run on CPU.

However, it uses external libraries for computation (such as oneDNN) that eventually rely on platform-specific instructions to run efficiently. As far as I know, WASM has incomplete support of these instructions, see e.g. https://github.com/WebAssembly/relaxed-simd/issues/9.

Currently we have no plans to explore this further, but any insight on this subject is welcome.

limcheekin commented 1 year ago

I found an answer from Bing Chat which might be relevant here, but not sure whether it is valid as I hadn't verified it: TLDR: PyTorch -> ONNX -> WASM

To convert PyTorch models to ONNX format, you can use the torch.onnx.export() function provided by PyTorch. Once you have an ONNX model, you can follow the steps outlined in the ONNX Runtime documentation to build ONNX Runtime for WebAssembly².

(2) Build for web | onnxruntime. https://onnxruntime.ai/docs/build/web.html.