search

webmachinelearning / webnn

🧠 Web Neural Network API
https://www.w3.org/TR/webnn/
Other
369 stars 46 forks source link

WebNN API is not support Mirrorpad, SquaredDifference, Pow, TransposeConv #108

Closed calvinlcchen closed 3 years ago

calvinlcchen commented 3 years ago

As a use case about style transfer, source is https://intel.github.io/webml-polyfill/examples/style_transfer/?prefer=none&b=WebGL&m=fast_style_transfer_onnx&s=image&d=0&f=WebNN, Some ops style transfer used is not be supported in current WebML API spec. Please add following ops to support list. thanks. mirrorpad, SquaredDifference, Pow, TransposeConv for tflite format models. mirrorpad, Pow, ConvTranspose for onnx format models.

huningxin commented 3 years ago

According to 29 Oct CG call, this is related to the style transfer use case and it would be good to support. Thanks for the proposal @calvinlcchen !

As a use case about style transfer, source is https://intel.github.io/webml-polyfill/examples/style_transfer/?prefer=none&b=WebGL&m=fast_style_transfer_onnx&s=image&d=0&f=WebNN,

The models are based on Fast Style Transfer in TensorFlow that is based on "a combination of Gatys' A Neural Algorithm of Artistic Style, Johnson's Perceptual Losses for Real-Time Style Transfer and Super-Resolution, and Ulyanov's Instance Normalization."

The InstanceNormalization could be defined as a high-level function that can be decomposed to add, sub, mul, div, reduceMean and sqrt. (The SquaredDifference and Pow are used to convert InstanceNormalization, we can skip them).

The mirroredpad can be supported by defining a general pad operation with symmetric padding mode.

The TransposeConv of tflite model and ConvTranspose of onnx model can be supported by defining a convTranspose2d

In the CG call, Chai also mentioned the Fast Neural Style Transfer in ONNX model zoo. This model also uses InstanceNormalization and Pad. Instead of using ConvTranspose, this model use Upsample. I found the Upsample is marked as deprecated in ONNX doc. Which operation should we use instead @wchao1115 ?

wchao1115 commented 3 years ago

There is an ONNX port of this model here. ONNX Upsample is replaced with Resize. This is one of the popular computer vision models that relies on InstanceNorm, webnn should support it. I'll look into adding the necessary operators.

huningxin commented 3 years ago

The transposed convolution was proposed by @wchao1115 in https://github.com/webmachinelearning/webnn/issues/60 and discussed in earlier CG call. The decision was the transposed convolution was not urgent at that time and could be supported later. Now given the style transfer models will be supported, it looks like a good time to add transposed convolution.

wchao1115 commented 3 years ago

Thanks @huningxin for digging that up. I did remember I recommended it, but we punted on it earlier on. I wonder if we want to revert that call now in order to support this model. As I mentioned during the CG call today, there are other models that expect transpose to also work.

anssiko commented 3 years ago

My guidance for the group is we may revisit and revise old resolutions given new information becomes available.

This is one such case. The group has received new information that style transfer has significant user demand, which motivates us to add support for this model. We also received new information on op support requirements. This is also one of the identified use cases.

Unless someone objects, I propose we move forward with this.

huningxin commented 3 years ago

@calvinlcchen , given #123 merged (thanks @wchao1115 ), could we close this issue?

wchao1115 commented 3 years ago

I believe we can now close this issue.

anssiko commented 3 years ago

@calvinlcchen we consider this issue resolved with the recent spec updates and thank you for your contribution!