Add concrete algorithm options like 'bilinear' and 'bicubic' to ctx.imageSmoothingQuality

[josephrocca]

I recently ran into a problem where a machine learning model expected images to be resized with bicubic interpolation (else lower accuracy), but there doesn't seem to be a way to specify a particular interpolation algorithm for a canvas context. The solution in the meantime for me will probably be to find a Rust implementation and port it to JS via wasm, but it would be great if this weren't necessary.

It seems plausible that there would be other types of use cases that would also need to use a particular resize algorithm, and I figure that the common resize algorithms will already exist "behind" the current low, medium, high settings anyway, so it might be easy enough on the implementer side of things to make it worth pursuing this.

Any thoughts on this?

Edit: I suppose this would also apply to createImageBitmap's resizeQuality option, which I've just discovered.

[pshaughn]

Right now, the interoperability guarantees about drawing on a canvas and then reading back the result as data are very weak. The spec leaves a lot of details open for implementers to be free to optimize and improve. Aside from interpolation, other unexpected details like colorspace conversion and subpixel font antialiasing can sneak up on you.

I think a canvas with better interoperability guarantees would be a great thing to have (maybe as a specially created type of OffscreenCanvas to avoid breaking what exists?), and precisely specified interpolation would be a crucial part of that.

[annevk]

cc @whatwg/canvas

[kdashg]

This is something I would really recommend doing yourself in webgl, in order to actually control things. (Or wasm +/- simd if you need low-latency access to it CPU-side) Resizes like this (like canvas2d in general) are supposed to be "good enough" for visual content, with allowances for UAs to choose what works best.

I'm surprised you're using canvas2d at all for ML. I wouldn't expect the guarantees it has would be portable enough. Is this just for resizes?

[josephrocca]

@kdashg I've used OffscreenCanvas for a few different ML processing tasks (resize, crop, various augmentations, extracting RGBA, etc.) before this, but this is the first time I've needed a very specific resize algorithm. I think "tiny" things like differences between resize algorithms affect some ML models more than others. In my use case the difference doesn't seem to be too significant, but there is certainly a difference in model behavior.

IIRC, TensorFlow.js uses canvas for image processing stuff - I remember creating/commenting-on an issue to request that they switch from canvas to OffscreenCanvas so it would work in Web Workers.

For now I've gone with wasm-vips and it seems to work well! Feel free to close this if it's very unlikely that this idea would get support. Would be neat if a library weren't required for this sort of thing, but I completely understand if this idea is unlikely to get traction for technical/practical/priorities based reasons.

[robertmazzo]

@josephrocca I would like to apply bicubic interpolation to our images when viewed in our application running under Chrome, specifically on zoom (where pixelation is clearly visible). From what I understand (as per the last comment at https://bugs.chromium.org/p/chromium/issues/detail?id=618324#c51), setting canvas imageSmoothingEnabled to true, and imageSmoothingQuality to 'high' - Bicubic interpolation is currently being used as the default. If I have misunderstood, would you kindly point me to a library which I can integrate into our custom image viewer? In other words, pulling the imageData off the canvas and passing it thru an interpolation function before redrawing to the canvas. - Thank you.

[josephrocca]

Hey @robertmazzo, if you're talking about the resizing algorithm used by canvas drawImage calls, then that's surprising because I tried the 'high' value and it didn't seem to resize bicubicly. In any case, since, as an API, it's not guaranteed to be bicubic (especially across browsers), I don't think it'd be a good idea to lean on that. Here's a simple bicubic resize function you can use/adapt:

https://gist.github.com/josephrocca/d97e0532f34e1205f4006d45ca909024

Please read the comments in that file - wasm-vips is still in the very early stages with a bunch of sharp edges (e.g. need to explicitly delete all images as shown else you'll get a memory leak), but the author is working on improving it for the 0.0.2 release, which should also get it's own versioned CDN url.

[robertmazzo]

Hi @josephrocca - Thanks for your reply. And yes I am referring to drawImage(), called specifically after we apply our own zooming function on the canvas. I will check out your js code above with regards to using wasm-vips. But also, I do notice a smoothing effect on my image when the 'high' setting is applied (i.e. it's no longer pixelated), but how would I know whether it's linear, bicubic or something else? For reference, I've posted a few images here at SO - https://stackoverflow.com/questions/71576232/interpoloation-algorithm-for-image-smoothing-angular-12-environment?noredirect=1#comment126526382_71576232