Could ImageDecoder support non-RGBA8 color formats?

[mcclure]

Is this the correct place to post feedback on the API standard?

WebCodecs seems to primarily exist to support videoconference applications. But, I am interested in it for a particular reason: I am creating WebGPU apps, and I could use ImageDecoder to load images into memory before uploading them as textures to the GPU. (You can have the browser load an image and send it directly to the GPU using ImageBitmap/copyExternalImageToTexture, and this is actually more efficient than routing through ImageDecoder, but imagine there is some sort of CPU-side preprocessing I want to do before creating the texture.)

ImageDecoder.decode produces a VideoFrame, which can copyTo pixel data into a buffer. The pixel data is formatted according to the VideoFrame's VideoPixelFormat. However, aside from RGB8 and RGBA8, the only other formats are video-centric color formats (YUV). Although ImageDecoder can be used to decode conventional image formats, it appears to be lacking support for some types color formats that browser image decoders handle without problems and which are relevant to my use case (WebGPU):

• Fewer than three channels. PNG supports 1, 2, 3 or 4 channels. WebGPU supports one, two, or four-channel textures (R8, RG8). Although this is not really a problem since I could trivially take in the RGBA or RGBX data from VideoFrame and discard the channels I don't need, there will be a performance concern; for a grayscale PNG, you will be taking the time to duplicate the one channel into four, and then I will have to add an extra step to copy the one channel into another buffer.
• More than 8 bits per color component. PNG supports 16-bit color channels, and so does WebGPU (WebGPU also supports some oddities such as 32-bit color channels and floating-point color channels, though I'm not sure if any web image format can convey this data).

For example, here is a PNG image containing one 16-bit grayscale channel (explanation here), and browsers (my copy of Chrome at least) are able to display this image fine. (I haven't investigated the color formats supported by webp, JPEGXL etc)

This might seem trivial because most computers can only display 8-bit color components anyway. But HDR10 displays are increasingly common, and WebGPU introduces a separate interesting use case (although again, with WebGPU you'd be more likely to use the ImageBitmap path) because you might create textures containing non-pixel data, IE, a texture might contain some sort of unusual material map which for some reason uses more than 8 bits of resolution, and that texture would be an input to a shader rather than being displayed directly. You could also imagine JavaScript applications that use PNG to compress non-image tabular data. At the moment, ImageDecoder is the closest thing that the browser has to a sensible "load image, return pixel data to CPU as an array" API, so it would be nice if it could support some of these diverse color formats that the browser itself can.

[dalecurtis]

The root issue for your first request is that we don't have R8, RG88, or RGB888 pixel formats. We could certainly consider adding them. WebCodecs tries not to silently transform content where possible, so in that spirit the UA could just emit decoded images in these pixel formats if they matched the default output type for the image.

There's also issue #92 which would aim to provide conversion during copyTo. We've tried to avoid any language which would force a UA to emit a certain format (since it may constrain power efficient options), so copyTo is probably the right path forward here once more pixel formats are available.

On your second point, issue #384 is relevant; in that it aims to add HBD pixel formats; which are required before we could output HBD image data without any conversion.

I'll leave this issue open for consideration of adding R8. RG88, RGB888 pixel formats.

[sandersdan]

You can have the browser load an image and send it directly to the GPU using [...] copyExternalImageToTexture, and this is actually more efficient than routing through ImageDecoder.

Is this actually more efficient than ImageDecoder + importExternalTexture()? I am surprised to find out that copyExternalImageToTexture() exists but supports a different set of sources than importExternalTexture().