msaroufim
commented
3 years ago Hi @JonathanSum it may take some time still to find a proper solution to your problem. We are discussing a C++ backend for torchserve but that's not out yet.
Another option would be to deploy your code using Pytorch Mobile in case that's enough https://github.com/pytorch/ios-demo-app or https://github.com/pytorch/android-demo-app
If you suggest ONNX: ONNX allows the Pytorch model to be deployed in javascript, so we can run the model in the browser frontend-only without using the backend CPU or backend GPU. But the problem with ONNX was different versions support different features. They upgrade the ONNX to be ONNX web allowing it to run all the features. I am not sure. I think it may still have the issue of not supporting all the features.
But we have Serve, which is the native or something from Facebook. The problem is, it is a Rrestful API, so it will need the backend CPU or GPU to run everything. If you have a small business having million of people doing image classifiers, it may be difficult for the cpu or gpu in the backend.
Pytorch discussion forum suggested using the CPU C++. It is true that we have webassbly to turn C++ code to be javascript binary to run in the browser. But that CPU may be slow, and it is about 200MB.
Thus, can we have something like a tiny version of Pytorch or Serve in the future that allows people to deploy Bert, GPT, Coloring, GNN, or any complicated model that requires GPU in Javascript for inference, not training only? The GPU version of Pytorch requires like couple GB to download. It may be fine to have a CPU version of Pytorch running it in Javascript because the C++ CPU of torch serve is huge, which is 200MB. I know ONNX Pytorch was built by webassbly, and I guess it may works if the super-smart Facebook team tries it.