search

WebKit / standards-positions

WebKit's positions on emerging web specifications
https://webkit.org/standards-positions/
240 stars 18 forks source link

Allow converting VideoFrame to RGB pixel formats by calling VideoFrame.copyTo() #341

Closed Djuffin closed 1 month ago

Djuffin commented 2 months ago

WebKittens

@youennf

Title of the spec

WebCodecs

URL to the spec

https://www.w3.org/TR/webcodecs/#dom-videoframecopytooptions-format

URL to the spec's repository

https://github.com/w3c/webcodecs

Issue Tracker URL

No response

Explainer URL

https://gist.github.com/Djuffin/9e2f98025ead49998524510cfeed8d33

TAG Design Review URL

https://github.com/w3ctag/design-reviews/issues/951

Mozilla standards-positions issue URL

https://github.com/mozilla/standards-positions/issues/1017

WebKit Bugzilla URL

No response

Radar URL

No response

Description

Converting YUV video frames to RGB is often required for processing them in libraries like TensorFlow.js and OpenCV.js. Previously the only possible way to achieve this was rendering the frame on a canvas.

Specifying VideoFrameCopyToOptions.format and VideoFrameCopyToOptions.colorSpace make it possible to asynchronously convert frames to RGB pixel format by calling VideoFrame.copyTo() without having to use an extra canvas.

youennf commented 2 months ago

The use case is fine and the API is better than going through a canvas (more straightforward and async by nature).

marcoscaceres commented 1 month ago

This is a draft position. Unless anyone objects within a week we will change it to the official WebKit position.

@Djuffin, we discussed this proposal and we are "supportive" with a concern around the API Design.

The one thing that would be a nice addition is having some means to synchronously detect if copying to a colorspace would be possible without first needing to call .copyTo() (i.e., some kind of support check). We can see it's feasible to, for example, create a VideoFrame and perform support checks by seeing if it throws... but, to us, that feels like not so great developer ergonomics.

Have you thought about adding some kind of mechanism to check what colorspaces the browser supports?

Djuffin commented 1 month ago

VideoFrame.allocationSize() is synchronous and it throws if conversion is not supported.

tomayac commented 1 month ago

Else, for inspiration, we just introduced ClipboardItem.supports('image/svg+xml') to the Clipboard API. A similar approach could be feasible here…

Djuffin commented 1 month ago

Yes, we should consider introducing VideoFrame.canCopyTo(), especially if we end up adding more format options to VideoFrameCopyToOptions

marcoscaceres commented 1 month ago

Note that WebIDL will drop dictionary member on the floor, so VideoFrame.canCopyTo() can't rely on dictionary (it could take an object though, and then iterate over the member names that way to reject on any unknown member name... and then check the details of each that is supported.)

marcoscaceres commented 1 month ago

Closing with "support" position. Please ping us on the W3C side when there's a PR for us to review!