This repository is for anyone interested to run PyTorch fast-neural-style example in web browsers. The performance is by no means optimal due to many workarounds for issues and limitations during the conversion process and different operator/layer support level between PyTorch and ONNX.js. But it serves the purpose to understand what it takes to go through the entire process.
It is an example for practicing and learning what it takes to make the PyTorch generated models portable to other deep learning frameworks. ONNX.js is set as the target deep learning framework as it's very new, hance still primitive.
This project is roughly based on the following open source projects:
This repo:
How-to guides on tweaking PyTorch fast-neural-style repo:
The objective is simple:
PyTorch FNS example --> PyTorch model files (.pth) --> ONNX model files --> ONNX.js on web browsers
There are many style transfer implementations. PyTorch's fast-neural-style example is the most facinating one. Partly due to the way it is implemented provides a much finer style-transfered images. To run the inference in browser, the following 3 major steps are taken:
Sounds straight forward!? Read on...
These steps may seem easy, but in practice it is way much more complicated.
The following were the few of major obstacles encountered during the process, just to give an idea on what are possible issues there may be:
Operator/layer support levels are very different.
nn.InstanceNorm2d()
is exported as ONNX InstanceNormalization()
, but not supported by ONNX.js.nn.functional.interpolate()
is exported as ONNX Upsample()
, but not supported in ONNX.js.Base tensor opset levels are different between PyTorch, PyTorch ONNX Export and ONNX.js
mean()
, along 1 axis. i.e. torch.mean(t, [2,3])
is not supported by PyTorch ONNX export. (Although both PyTorch and ONNX.js supports multi-axis reduction ops.)ONNX.js has quite a few issues.
Reshape()
, which is converted from PyTorch's view()
.pow()
+ mean()
produces NaN
values in javascript.pow()
op is very buggy.
Dynamic tensor shapes exported by PyTorch ONNX is very large and hogs memory like hell.
Reshape
and Gather
ops). Although still works, it is not practical to use such model files in web browsers.ONNX.js support for Mobile devices, such as Android, is still not stable.
mosaic zero-pad
as first model as some mobile devices does not run pad
op correctly using webgl
backend. All models should work correctly on desktop browsers.For more details on tweaking and working around the differences between PyTorch, Exporter and ONNX.js, please see PyTorch fast-neural-style for web.
node.js
ONNX.js can be served locally by node.js
via npm
.
Windows npm
installer:
https://nodejs.org/en/#download
Ubuntu npm
installation:
sudo apt install nodejs npm
Install node.js http-server module
npm install http-server -g
Run node.js server locally:
http-server . -c-1 -p 3000
Open a browser and go to the following URL:
http://localhost:3000