Processing pipeline: Create VideoFrame out of WebGPU result

[tidoust]

A video processing pipeline may have multiple frame processing steps, some of which could be done using WebGPU. The result of the WebGPU processing may need to be further processed, e.g. to feed it into WebCodecs' VideoEncoder and send the encoded stream to another peer using WebTransport.

Importing a VideoFrame to WebGPU as an external texture is now relatively straightforward. What is the most efficient way to get a VideoFrame back from WebGPU when the processing is over?

See Experimenting with video processing pipelines on the web for sample code that uses WebGPU as part of a processing pipeline (WebGPU code in VideoFrameTimestampDecorator.js). The sample code waits for onSubmittedWorkDone and creates a VideoFrame out of the rendered <canvas>. This approach seems to introduce a ~10ms delay in average and it is not clear whether this is just a temporary implementation hiccup (support for WebGPU and WebCodecs in Chrome are still under development) or just not the right approach to creating a VideoFrame (the shaders can be drastically optimized for sure, but the delay seems to appear even when the shaders merely sample the texture).

In theory at least, using a <canvas> seems unnecessary (no need to actually display the result) but a VideoFrame cannot be constructed out of a GPUBuffer.

[tidoust]

The sample code waits for onSubmittedWorkDone and creates a VideoFrame out of the rendered . This approach seems to introduce a ~10ms delay in average and it is not clear whether this is just a temporary implementation hiccup (support for WebGPU and WebCodecs in Chrome are still under development) or just not the right approach to creating a VideoFrame (the shaders can be drastically optimized for sure, but the delay seems to appear even when the shaders merely sample the texture).

I still don't know why this approach introduces a ~10ms delay, but I now know that this approach is not the right one ;)

There is simply no need to wait for onSubmittedWorkDone in practice. The browser takes care of ensuring that all submitted work has been completed before it lets the application access the resulting canvas, as discussed in https://github.com/gpuweb/gpuweb/issues/3762#issuecomment-1398339650

In other words, the most efficient way to a VideoFrame back from WebGPU when the processing is over is simply to create a new VideoFrame out of the canvas associated with the WebGPU processing. No need to worry about synchronization issues, the browser handles it under the hoods.