fintelia
commented
1 year ago I would suggest going with option (1). It might be possible to get (2) to work, but due to API mismatches I don't think it is a good fit.
In the image crate, an instance of an ImageDecoder is constructed via calling its new method. For instance, TiffDecoder::new. That's how you specify the bytes of the image you want to decode. Right now, the new method isn't actually part of the trait, but that'll likely change in the next major version.
But the bigger problem is around wanting to get image metadata before decoding the image itself. The assumption is that new only parses the image metadata, after which you can set decoding limits (set_limits), query image properties (dimensions, color_type), and then only after those are complete start the actual decoding (read_image). Without that capability, there isn't much point in implementing the ImageDecoder trait.
mcclure
seanaye
TLDR: "I am going to make a crate built atop Image. Would it make more sense to make it as a PR to Image instead?"
I would like to be able to...
The Image crate has really convenient container types and utilities. Once I have decoded an image, I would like to represent it using Image's types.
However, rather than using Image's builtin decoders, I would like the decoding to be done by the browser using wasm-bindgen.
My specific use case for this functionality is ...
Space savings.
I have a small app that embeds a number of small PNG images (like <20kb worth). The app is dual-mode desktop/web; it draws using WebGPU, and can be built either using normal cargo or using wasm-pack. I initially used
imageto decode the PNGs. This took up a great deal of disk space. Using features to select onlyimage's PNG decoder and no others saved like 600k, but I could still do better.I used #[cfg] to replace the
image::load_from_memorycall, when run in wasm, with a wasm-bindgen load using ImageBitmap. This plus deselecting the png feature reduced the size of my final optimized wasm from 380,553 bytes to 262,596 bytes. I think that space/load time are absolutely critical for webpage-embedded apps, so I would consider 117 kilobytes a very significant savings! When you consider the browser decoder supports not just PNG but also JPEG, webp and potentially other things as well, the potential savings is even greater.Draft
As mentioned, I have already implemented a browser-based memory->
image::ImageResult<image::DynamicImage>decoder, and I would like to publish the code as a crate (once I have added error handling). However, I can imagine three ways to go about this:Create a crate that exposes a function that returns a
image::ImageResult<image::DynamicImage>.Create a crate that exposes an ImageDecoder. This would be nice because hypothetically more of the ordinary Image API could be used(?).
However, I can't… figure out how to do this. There is a https://docs.rs/image/0.24.6/image/enum.DynamicImage.html#method.from_decoder method, but the description is unclear ("Decodes an encoded image into a dynamic image.". What?) and I don't understand "where you put in the image", IE, there is no way to specify the bytes or path of the image. There's no example code for this feature. Is it intended that clients of the
imagecrate be able to add their own custom decoders using ImageDecoder?Create a PR to
imagewhich adds a feature which replaces all decoders with the browser decoder.This might be possible someday, but is not possible now (see below)
There are two big problems that prevent (3) or even (2) here from being possible. One will fix itself in future web browsers, but the other would require
imageto change.The ImageDecoder problem
In the HTML standard, if you want to decode an image in code, you have two choices. The best way is to use ImageDecoder, which is part of the WebCodecs draft standard, Unfortunately this class is currently only supported in Chromium browsers, not Firefox or Safari.
In the absence of ImageDecoder, you have to go through a rigamarole of creating a Canvas, creating an ImageBitmap, drawing the ImageBitmap to the Canvas, and then reading back the pixel data from the Canvas. This creates several problems.
First off, the data is alpha premultiplied, and you cannot opt out of this. (It also appears to be RGBA8-only, but this seems to also currently be a limitation of ImageDecoder…)
Second off, the temporary Canvas is a potentially large resource expenditure, and making this efficient would probably require API changes that mismatch with Image. In my current browser implementation, I have the user create an object which holds the scratch Canvas and release it when they're done decoding (for example imagine loading 16 large images in a row, this approach will mean leaving 1 temporary canvas for the GC to clean up instead of 16…). There is no way I can see to mesh this with the
imageAPI (you could create a shared global canvas, but then you couldn't ever release it).However, the browser approach still doesn't mesh with Image's API, because
The async problem
Both the future ImageDecoder, and the current Canvas/ImageData path, require async calls. The image API does not expose a path for async image decoding, and in Rust's wasm target it is as far as I know impossible to call async methods within non-async ones (the
block_onfunctionality available on regular platforms does not work on wasm). However, if #1397 were ever revisited, then this would create new options.Do you have any thoughts? Might a PR of this type be worth attempting someday? Is there a way to make my "Plan (2)" above work?