[Feature request] Image-to-image (super-resolution)

[josephrocca]

Name of the feature Image super-resolution.

Reason for request I have a web app where people can generate images using a server-side stable diffusion model, and although it'd of course be possible for me to do the upscaling on the server, it'd be handy if I could allow them to upscale using their own compute - mainly to simplify my server-side deployment/orchestration stuff.

Additional context A while ago I got SwinIR working in the browser, and IIRC it was simple to get it working with ORT Web, but it takes quite a while to initialise due to the ONNX export resulting in 40k ops for some reason. This was probably a mistake in the way I was exporting, or a weird quirk of the specific export approach that I happened to use.

Size/speed notes:

• The SwinIR Medium x4 model is about 50mb (unquantized) and inference takes about 4 seconds for each 64x64 pixel patch on the CPU backend, so it's a bit slow for CPU on large images, but I'm guessing WebGL and WebGPU will make it viable.
• The PlayTorch demo provided in the SwinIR repo also takes about 4 seconds, although it doesn't say how big the image is - it does look like it's 64x64 based on the blurriness. So I'm guessing that demo also uses a CPU backend. Either that, or it's a 128x128 image and they're using a GPU backend. But just guessing here.
• This replicate demo runs on an A100, and it takes 5 seconds for a 512x512px image, which is pretty good. For reference, ORT Web CPU backend takes about 5 minutes for a 512x512 image. Even if WebGPU (on an average laptop or whatever) could only get it down to a minute or so, it would be quite useful I think.

That said, there might be better models available now - according to the docs, Hugging Face Transformers supports these:

• Swin Transformer (from Microsoft) released with the paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows by Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo.
• Swin Transformer V2 (from Microsoft) released with the paper Swin Transformer V2: Scaling Up Capacity and Resolution by Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo.
• Swin2SR (from University of Würzburg) released with the paper Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration by Marcos V. Conde, Ui-Jin Choi, Maxime Burchi, Radu Timofte.

[josephrocca]

(Also: The above-mentioned SwinIR demo crashes my whole [Ubuntu] machine on Chrome. It didn't originally do this. Works fine on Firefox. If someone with a Windows/Mac machine could test it [after saving all your work] so I know whether to report it to the Chromium devs, that would be handy. Might just be a weird Ubuntu thing.)

[xenova]

We'll definitely be adding image-to-image pipelines once WebGPU is supported. 👍

[josephrocca]

:eyes: https://twitter.com/NielsRogge/status/1709259362143707553 :eyes: