Mutable ImageData like JavaScript's ImageData

[s1gtrap]

Summary

I'm really confused about the ImageData struct. Is it meant to be read-only?

An example from MDN ought to also be possible with WASM, right?

const imageData = ctx.createImageData(100, 100);

// Iterate through every pixel
for (let i = 0; i < imageData.data.length; i += 4) {
  // Percentage in the x direction, times 255
  let x = (i % 400) / 400 * 255;
  // Percentage in the y direction, times 255
  let y = Math.ceil(i / 400) / 100 * 255;

  // Modify pixel data
  imageData.data[i + 0] = x;        // R value
  imageData.data[i + 1] = y;        // G value
  imageData.data[i + 2] = 255 - x;  // B value
  imageData.data[i + 3] = 255;      // A value
}

// Draw image data to the canvas
ctx.putImageData(imageData, 20, 20);

But obviously, since ImageData::data(&self) produces a Clamped<Vec<u8>> the data is copied to a fresh Vec from the underlying Uint8ClampedArray and not writable like in the JS example above, so unless I misunderstood something entirely?

Additional Details

I'm hoping to make calls to WASM for somewhat efficient rendering but having to allocate a new ImageData every frame sort of defeats the purpose. Knowing how ImageData worked on the JS side of things I was expecting to be able to modify one allocated beforehand like so

#[wasm_bindgen]
pub struct Handle(web_sys::CanvasRenderingContext2d, web_sys::ImageData);

#[wasm_bindgen]
pub fn init(ctx: web_sys::CanvasRenderingContext2d) -> Handle {
    let id = ctx.create_image_data_with_sw_and_sh(1.0, 1.0).unwrap();
    Handle(ctx, id)
}

with something like

#[wasm_bindgen]
pub fn step(h: &Handle) {
    log::debug!("step {}x{}", h.1.width(), h.1.height());
    h.1.data().0[3] = 255;
    log::debug!("{:?}", h.1.data()); // Clamped([0, 0, 0, 0])!?
    h.0.put_image_data(&h.1, 0.0, 0.0);
}

So is it possible to modify the underlying byte array? I know the data field is marked as read-only but modifying said data isn't.

(I see a reference to Clamped<&mut [u8]> on the page for Clamped but as far as I can tell it's not possible to get from an ImageData)

[s1gtrap]

Just realized I could pass imageData.data as &mut [u8] 😂

I assume there's nothing to be gained from calling ctx.putImageData from Rust (and there's no copying when passed with &mut [u8]), so I guess that sort of settles my problem?

I'm still wondering why the only way to access ImageData.data is by copy, and why there isn't a 'mutable getter' though.

[peter9477]

Can you clarify whether you got something like this to work, so you could modify the data? I don't follow your explanation from Apr 21 about what you realized you could do, or what settled your problem. It seems like you're saying you can't do this from Rust via ctx.put_image_data(). I've been trying similar things and so far my image data is effectively read-only.

[s1gtrap]

Sure. I forgot what I ended up doing exactly, but what I meant by that was I realized it was possible to pass a Uint8Array from JS-land to a #[wasm_bindgen] fn taking a &mut [u8] and mutating it that way. Then, back in JS, call imageData.data.set(newData) to swap it and ctx.putImageData(imageData, 0, 0) to copy it to the actual frame buffer.

I learned that there was no trickery to be had with the double buffering and that you had to copy the image data over, but it's been working okay for me so far.

[peter9477]

Thanks for that. I've ended up for the moment with a different approach, where I'm repeatedly modifying a stored Vec<[u8]>, and then doing let imdata = ImageData::new_with_u8_clamped_array(Clamped(mydata.as_slice()), 0) followed by ctx.put_image_data(&imdata, 0.0, 0.0) in the Rust/wasm code, saving me a trip up to JavaScript. I have no idea yet if that's more costly than the other way, or cheaper, or a toss-up, though the performance appears quite acceptable for my case.

[petalas]

@alexcrichton @s1gtrap @peter9477

Hello, could you help me out with a vaguely related issue please? If not I can open a new question.

I have an HtmlCanvasElement, associated CanvasRenderingContext2d and ImageData. All created in rust.

I need to do some calculations with the pixel values after using the context to draw.

The problem is, calling methods such as .fill or .fill_rect using the context does not seem to modify my ImageData.

It seems get_image_data is returning a 'snapshot', how do I maintain a reference to the underlying vector? I'm really hoping even if it's not possible in JS there's a workaround on the rust side of things.

I have also attempted storing a Vec\<u8> (by calling .data().to_vec()) but had the same issue.

My current workaround is to keep replacing my ImageData with a fresh copy by calling context.get_image_data after drawing, but this is very costly (in a loop, part of a rendering engine I'm working on).

TLDR: how do I avoid having to constantly call get_image_data?

Here's a very simplified example with a 1x1 canvas:

let testCanvas = create_canvas();
resize_canvas(&testCanvas, 1, 1);
let testCtx = get_context(&testCanvas);
let testData: ImageData = testCtx.get_image_data(0.0, 0.0, 1.0, 1.0).unwrap();

console::log_1(&format!("Test data before : {:?}", &testData.data().to_vec()).into());
testCtx.set_fill_style(&JsValue::from("#fff"));
testCtx.fill_rect(0.0, 0.0, 1.0, 1.0);

// Shouldn't have to do this, I expect testCtx.fill_rect to have modified testData
// FIXME: how to avoid calling calling get_image_data?
testData = testCtx.get_image_data(0.0, 0.0, 1.0, 1.0).unwrap();
console::log_1(&format!("Test data after : {:?}", &testData.data().to_vec()).into());

fn create_canvas() -> HtmlCanvasElement {
    let window = web_sys::window().expect("no global `window` exists");
    let document = window.document().expect("should have a document on window");

    document
        .create_element("canvas")
        .unwrap()
        .dyn_into::<HtmlCanvasElement>()
        .unwrap()
}

fn resize_canvas(canvas: &HtmlCanvasElement, w: u32, h: u32) {
    canvas.set_width(w);
    canvas.set_height(h);
}

fn get_context(canvas: &HtmlCanvasElement) -> CanvasRenderingContext2d {
    let opts = js_sys::Object::new();
    js_sys::Reflect::set(&opts, &"willReadFrequently".into(), &true.into()).unwrap(); // not sure if this works either

    canvas
        .get_context_with_context_options("2d", &opts)
        .unwrap()
        .unwrap()
        .dyn_into::<web_sys::CanvasRenderingContext2d>()
        .unwrap()
}

Ouput:

Test data before : [0, 0, 0, 0]
Test data after : [0, 0, 0, 0] // without copying over
Test data after : [255, 255, 255, 255] //  only works if I overwrite testData = testCtx.get_image_data

Is this because I need to be storing a &ImageData instead of ImageData? If so, how to deal with the lifetime parameter?

#[wasm_bindgen()]
pub struct Engine<'a> { // error: structs with #[wasm_bindgen] cannot have lifetime or type parameters
    ...
    testData: &'a ImageData,
}

Apologies if I'm missing something obvious, I'm new to rust.

[s1gtrap]

If I understood it correctly you intend to edit what is on screen by directly editing the ImageData as returned from getImageData?

I'm afraid it wouldn't work like that as the ImageData you get is not a live slice into the frame buffer: instead it's copied with getImageData. If I recall correctly by reading up on current thread as well as my StackOverflow question you can't avoid copying back and forth either way as putImageData also copies.

Do you need to use the CanvasRenderingContext2d for rendering? I know your example is a simplification and all, but can you get away with modifying it in Rustland instead? Supposedly getImageData is the 'slow one':

From your description you can indeed remove entirely the getImageData step, keep only one ImageData object, created either woth the new ImageData(w, h) constructor or ctx.createImageBitmap work on that and putImageData it on the canvas context every frame. And the good thing is that getImageData os really the slow one.

https://stackoverflow.com/questions/71917851/fast-double-buffered-data-display-with-simultaneous-iteration-for-html5-canvas#comment127083342_71917851

[petalas]

Not on screen but yes, I want to maintain a 'live' reference to the underlying buffer, and modify it by calling methods on the associated context.

I think you're right in that getImageData returns a snapshot (very unfortunate if there's really no way around that).

I'm really hoping that at least on the rust side of things there's a way (even unsafe would be fine) to get a reference to the underlying vector, is there a reason why that would not possible?

Implementing my own drawing logic and manipulating a Vec<u8> directly could work in theory but would be a major pain (rendering overlapping semi-transparent polygons).

The result looks like this (the 'Generated' one in the middle):

Evolving new 'drawings' (was going to be a genetic algorithms project but ended up more like simulated annealing.).

Anyway, I'm generating mutations (randomly shifting points, colors, order of polygons etc), I have a fitness function to calculate 'distance'/error vs the reference image, in a canvas that is not visible (not even added to the DOM) and only update the visible canvas when I find a 'better' mutation (can be very rare, most mutations never get displayed on the UI).

In theory I should even be using an OffscreenCanvas and web workers but haven't gotten that far yet.

Open to suggestions on how to speed things up but the majority of the time is taken up by get_image_data currently, massive bottleneck.

[s1gtrap]

I'm really hoping that at least on the rust side of things there's a way (even unsafe would be fine) to get a reference to the underlying vector, is there a reason why that would not possible?

Yea I highly doubt you'd get away with that for the time being (interesting proposal over at whatwg/html#5173). It's not even a rust thing, just a matter of limited memory access for obvious reasons.

Looks like you need to reconsider your need for canvas rendering along with rust data access as the two seemingly don't play nice together.

[petalas]

Yea I highly doubt you'd get away with that for the time being (interesting proposal over at whatwg/html#5173). It's not even a rust thing, just a matter of limited memory access for obvious reasons.

I see, that's unfortunate.

Looks like you need to reconsider your need for canvas rendering along with rust data access as the two seemingly don't play nice together.

Any good options other than implementing my own fill_rect? 😕

[s1gtrap]

Any good options other than implementing my own fill_rect? 😕

Afraid not.. At least not from me at the moment. In the end I opted for copying from rust with putImageData (and a few extra steps) and got distracted. I'm glad I'm not the only one mad enough to be dealing with this sort of issue though.

[Dudly01]

Just realized I could pass imageData.data as &mut [u8] joy

I assume there's nothing to be gained from calling ctx.putImageData from Rust (and there's no copying when passed with &mut [u8]), so I guess that sort of settles my problem?

I'm still wondering why the only way to access ImageData.data is by copy, and why there isn't a 'mutable getter' though.

Sure. I forgot what I ended up doing exactly, but what I meant by that was I realized it was possible to pass a Uint8Array from JS-land to a #[wasm_bindgen] fn taking a &mut [u8] and mutating it that way. Then, back in JS, call imageData.data.set(newData) to swap it and ctx.putImageData(imageData, 0, 0) to copy it to the actual frame buffer.

I think not all your steps are needed. In JS, assigning imageData.data to a variable will be a reference, if I am not mistaken. Then, using your &mut [u8] argument type idea, a Rust function can take that variable and modify it. As a reference was modified, no need to "call imageData.data.set(newData) to swap it", only call ctx.putImageData(imageData, 0, 0).

Here is some code snippets that I put together using these ideas. It worked on my machine (TM).

JS only code

```javascript const draw = () => { let imageData = canvas.getContext('2d').getImageData(0, 0, canvas.width, canvas.height); let pixelData = imageData.data; // Iterate through the pixels of the RGBA image. for (let i = 0; i < pixelData.length; i += 4) { // No need to draw on all if ((i / 4) % 10 === 0) { pixelData[i] = 0; // Red component pixelData[i + 1] = 255; // Green component pixelData[i + 2] = 0; // Blue component // The alpha component (pixelData[i + 3]) remains unchanged } } canvas.getContext('2d').putImageData(imageData, 0, 0); }; ```

Rust + WASM

Same as pure JS, but the `pixelDatä` is modified by WASM written in Rust. ([I am not using a bundler, hence the initialization.)](https://rustwasm.github.io/wasm-bindgen/examples/without-a-bundler.html) ```javascript import init, * as wasm from './pkg/my_package.js'; async function drawWasm() { // Instantiate the WebAssembly module await init("./pkg/net_bg.wasm"); let imageData = canvas.getContext('2d').getImageData(0, 0, canvas.width, canvas.height); let pixelData = imageData.data; wasm.draw(pixelData); canvas.getContext('2d').putImageData(imageData, 0, 0); } ``` The Rust part with your `mut& [u8]` trick. ```rust #[wasm_bindgen] pub fn draw(pixels: &mut [u8]) { let count = pixels.len(); for i in (0..count).step_by(4) { if (i / 4) % 10 == 0 { pixels[i] = 255; // Red component pixels[i + 1] = 255; // Green component pixels[i + 2] = 0; // Blue component } } } ```

However, it feels to me that the documentation of Clam suggests that the input type of the rust function should not be &mut [u8], but &Clamped<&mut [u8]>. But that will error with the trait boundClamped<&mut [u8]>: RefFromWasmAbiis not satisfied. So I am gonna keep using the former, until I encounter issues.

All in all, thanks for your input. I can now finally progress with my project.