Polyfill loading in Web Worker

[waldobeest]

Hi ennuicastr and Yahweasel,

Firstly, thanks so much for the work you guys are putting in to get this library and the underlying libav.js in such a useable state.

I have a somewhat unique usecase for the polyfill, which I am hoping you can assist me with.

Similar to the video-decoder-vp8-opus example, I am demuxing a webm vp8 video in the browser to canvas.

Due to performance struggles, I have successfully moved most of the workings to a web worker, which then sends back the VideoFrame (without the polyfill) and the LibAVWebCodecs.VideoFrame (with the polyfill).

What I have noticed is that displaying the VideoFrame from web worker to main process is that I need the polyfill for both, calling the

const image = await LibAVWebCodecs.createImageBitmap(v[idx++]); here

In Chrome using WebCodecs, it renders as expected:

In Safari, I am unable to use that, with the exception:

When I change the image rendering code as such:

      const imgData = context.createImageData(w, h);
      imgData.data.set(v[idx]._data);
      context.putImageData(imgData, 0, 0);

Then I get some visual representation, but seemingly something is off

The Safari web worker instantiates the LibAVCodecs with:

   this.LibAVWebCodecs.load({
        noworker: true,
        libavOptions: {noworker: true}
    })

Any ideas on why the polyfill is not allowing the rendering properly? Is there a way to use context.createImageData effectively to solve this problem?

[Yahweasel]

I must admit, I'd never tested the polyfill on Safari. Most of it is just data pushing, so there's no reason it wouldn't work, but this step is exactly the kind of conversion step that could go wrong.

The result you're getting is half very strange and half slightly expected. It looks like YUV420P data (full size mono image with half-size color data). If you didn't convert the frame to RGBA, that's exactly what you should get. What's strange is... what caused it to be monochrome? It should be essentially random colors, since each pixel represents four input pixels. If you convert the data to RGBA first, I bet it'd work.

Either way, thanks for bringing the createImageData issue to my attention. I wasn't aware that it wasn't working on Safari. I doubt it will be difficult to address.

[Yahweasel]

Out of curiosity, what version of Safari are you using? The video-decoder-vp8-opus sample works great for me on Safari 16.3.

[waldobeest]

Thanks, Safari 16.3 works on your example yes. That's the weird part.

It stops working in both Safari and Chrome (when you force it to use LibAVWebCodecs.VideoDecoder) when you load the code in a web worker.

In a web worker, several objects are not defined, such as window and document.

Within Safari, the web worker has even less, and cannot use CanvasRenderingContext2D as you can see from the above Safari error. This means there is no background way to actually render the canvas in the web worker.

Let me see if I can package a small demo site with your example which shows this.

[Yahweasel]

In retrospect it's now fairly obvious why this is happening:

https://github.com/ennuicastr/libavjs-webcodecs-polyfill/blob/ae457dd29e96a8694450789eb0c329df20169c60/src/rendering.ts#L78

I use an instanceof check, but of course, if this frame has been transferred from another worker, it won't be an instance of (our) VideoFrame. If you can compile libavjs-webcodecs-polyfill, I bet that what would fix this is replacing all the " instanceof vf.VideoFrame"s in render.ts with "._data". I'll implement a better typechecker than that either this afternoon or Wednesday.

[waldobeest]

It seems the origCanvasDraw is not defined when called by:

const image = await LibAVWebCodecs.createImageBitmap(v[idx]);
context.drawImage(image, 0, 0);

When I change that to:

const image = await LibAVWebCodecs.canvasDrawImage(context, v[idx], 0, 0);

Error changes to:

[Yahweasel]

Did you load the polyfill (call load()) on the host or only the worker?

[waldobeest]

Yes, it is loaded in both.

I am making a branch on a fork of the library with a web worker example, where the issue is hopefully obvious. Taking a bit of time to load it correctly through CORS etc.

[Yahweasel]

The reason I ask is because origCreateImageBitmap and scalerSync will both be null if the library hasn't been loaded, so that looks a lot like an unloaded polyfill :)

[waldobeest]

I see now my conditional loading could have been clearer, seems I didn't load the polyfill in both places. I changed it now, and am now getting an error similar to what you were stating earlier about the RGBA and yuv420p.

So using:

const image = await LibAVWebCodecs.canvasDrawImage(context, v[idx], 0, 0);

[Yahweasel]

Yeah, this looks like things not transferring correctly from worker to host. I think only the data is making it, not the prototype, so the VideoFrame is incomplete. This can be fixed, but it will involve a bit of twiddling. In the short term, you could change that to just use _data instead of _libavGetData(). It looks like on Safari that method is lost.

[waldobeest]

Thanks, seems I'm sorted after looking into the _libavGetData() and ._data as you mentioned. Here is an MR for my local build if that helps you at all: https://github.com/ennuicastr/libavjs-webcodecs-polyfill/pull/8

[Yahweasel]

Thanks!

[Yahweasel]

Merged