Improve handling of ‘raw’ pixel buffers and ImageData

Description

Based on #113

This PR adds a new functionality to loadImage() and Image to load images from already decoded pixel buffers. Currently there is no way to do that and skia-canvas only supports loading images from encoded formats and does encoding on its own using Skia methods.

There are many cases when loading raw pixel buffers are beneficial. For example, when reading images from web cams, reading video frames from ffmpeg or while using custom image decoders that Skia doesn't support.

Details

In this implementation I added a new optional parameter called ImageOptions that currently supports only one optional property called raw of type ImageData. API consumer can provide this object to loadImage() function and to Image constructor. If colorType and passed image resolution doesn't match pixel buffer data, a regular Error will be thrown using existing callbacks.

Example with loadImage():

let img = await loadImage(pixelBuffer, {
  raw: {
    width: 1920,
    height: 1080,
    colorType: 'rgba'
  }
})
ctx.drawImage(img, 100, 100)

Example with Image constructor:

let img = new Image({
  raw: {
    width: 1920,
    height: 1080,
    colorType: 'rgba'
  }
})
img.src = pixelBuffer
ctx.drawImage(img, 100, 100)

In Rust code I created a new function called load_pixel_data that is responsible for creating SkImage. It doesn't align perfectly with existing code that uses getters/setters and I didn't find a suitable way of how to make it look better. May be we can remove set_data setter and make it a function as well but I didn't want to do that without consulting with you first.

Also currently only the most common pixel formats are supported (such as rgba, rgb, bgra and argb) and I'm not sure if we need to expose other less popular formats that Skia supports.

Tests

I added several tests that test different loading scenarios and also tested internally using different popular libraries that allow raw pixel buffer extraction (for example sharp and ffmpeg). Based on my simple benchmarks I didn't find anything unusual, CPU and RAM usage was normal.

Hi Gleb,

Just wanted to say thanks for the PR, which I pulled into my own fork and successfully tested.

I also added a counterpart feature to export the whole canvas as raw data asynchronously. It's sort-of like context.getImageData() but for the whole canvas and using worker pool threads (like toBuffer()/etc do). The result can be returned as Buffer or ImageData types or written to file as usual.

Also added a feature to specify a crop rectangle when exporting in any raster format or raw.

I'm using all this in a workflow that reads in various images formats with sharp, resizes and caches them in memory. Then later we draw them to a Skia Canvas (possibly with other elements, transforms, etc), export the canvas back to sharp which does final PNG compression (and possibly tiling into multiple parts) and then the resulting data is sent to a network client. We use sharp for compression because it (or rather libvips) is way, way better at it than what we can currently get out of skia-canvas (which of course has no compression options for PNG at all).

The drawing/compression/sending part can happen very often with multiple images possibly being requested in the same millisecond with refreshes at 10Hz or more being common. So, performance matters. Switching to raw processing, skipping unnecessary encoding steps, has been benchmarked as ~400% overall improvement in speed, w/out any extra CPU load or memory use. Nice win.

Anyway, it's all published, and documented in the changelog and readme if anyone cares to take a look. There are 2 branches which contain the core changes, though one builds upon the other.

I should mention that this is my first ever Rust project, so quite likely there are some prettier or more efficient ways of doing some things. The syntax and some concepts have a bit of a learning curve!

Cheers, -Max

Oh I forgot to add one caveat I found when loading raw pixel data to SkImage. Skia doesn't support decoding 3-channel/24-bit color pixels like RGB. It can only handle 1, 2, 4 or 8 byte formats (reference). So when trying to "raw-load" some PNG I had which was RGB only, Skia returns nothing.

ColorType.bytesPerPixel() returns 4 for RGB888x image type (whereas, for example, sharp properly shows such images as having only 3 channels).

Luckily sharp has an ensureAlpha() function which will add an alpha channel if it's missing to round the pixel data out to 32 bits in these cases. But it's something to watch out for.

Thanks so much to both of you for getting everything working on the raw-image front! I've tried to integrate the work from your branches to allow for un-encoded images to be used while trying to minimize the amount of new, non-standard syntax and behavior. Since the ImageData class already exists for this purpose, extending that rather than creating a new kind of raw-buffer-backed Image made more sense to me.

Enhancements to ImageData

The browser version of this class is always in 4-byte RGBA mode, but does take an options arg (which is currently just used for colorSpace).

Note for the future: it would be great to be able to support non-srgb colorspaces like display-p3 but rust-skia doesn't currently allow for it (though it may someday?).

I followed your lead in adding an optional {colorType:"…"} argument when creating it either via the constructor or the context's createImageData & getImagedata methods. I've also kept the aliases you chose for rgba, rgb, and bgra, but I wasn't sure the color type you mapped to argb (ARGB4444) was actually what it sounded like (since the alias name suggests another 4-byte reordering). Is this 2-byte format actually commonly used?

The new read-only .colorType & .bytesPerPixel attributes on ImageData instances reflect the colorType selected when initializing the object.

I've also added an asynchronous loadImageData function that mirrors the existing loadImage function (and shares the same fetching logic, now pulled into the internal fetchData function).

Enhancements to the Context's handling of ImageData objects

According to the spec, the only way to get an ImageData object drawn to the canvas is to use the putImageData method, which simply copies the pixels to a location, ignoring any transforms or filters that might be in place. To supplement this, I've updated the drawImage() & createPattern() methods to accept ImageData arguments as well as Image and Canvas objects.

The export methods (saveAs, toBuffer, etc.) now support a format arg/file extension of "raw" and an optional colorType argument which only applies for raw exports (if omitted, it defaults to rgba).

Speedups for `getImageData()` & Image loading

The getImageData method now uses the GPU (if enabled) and caches the canvas contents between calls so fetching additional ImageData rects from the canvas after the first one is now effectively 'free'. The cache stays valid until a new drawing command is issued.

The src-setting behavior on new Image objects now uses asynchronous i/o when loading local files. As a result, it's now necessary to await img.decode() or set up an .onload handler before drawing it, even if the src was non-remote. The Image class is now an EventEmitter subclass so .on("load")/.on("error") (as well as .off an .once) now work as well.

Roads not taken

Until I get some feedback about how broadly useful these raw-image-handling features are I didn't want to go too crazy with adding new APIs. So for the moment I've held back from adding some of the additional conveniences that you've both explored, including:

the premultiplied flag for exports (since it seems like Sharp can work with either through a flag of its own)
the shorthand .raw export properties for Canvas
a toImageData exporter
crop rects for exports (since these can already be accomplished through existing means; does the faster version of getImageData address this scenario for you @mpaperno?)

My highest priority for now is making it easier to interoperate with libraries like Sharp. Is there anything I've left out that seems essential for that use-case?

Thanks again to you both for these terrific contributions!

samizdatco / skia-canvas