WebXR Raw Camera Access API

mozilla / standards-positions

https://mozilla.github.io/standards-positions/

Mozilla Public License 2.0

639 stars 69 forks source link

WebXR Raw Camera Access API #667

Open bialpio opened 2 years ago

bialpio commented 2 years ago

Request for Mozilla Position on an Emerging Web Specification

Specification Title: WebXR Raw Camera Access Module
Specification or proposal URL: https://immersive-web.github.io/raw-camera-access/
Caniuse.com URL (optional): https://caniuse.com/webxr (the Raw Camera Access Module extends WebXR Device API)
Bugzilla URL (optional): https://bugzilla.mozilla.org/show_bug.cgi?id=1419190 (the Raw Camera Access Module extends WebXR Device API)
Mozillians who can provide input (optional):

Other information

The WebXR Raw Camera Access Module extends the capabilities of the core WebXR Device API by allowing the sites to request "camera-access" feature when creating XR sessions. The feature allows the sites to obtain access to camera pixels (via an integration with WebGL, exposing the pixels as WebGLTexture). This capability has been requested for a long time by developers.

ekr commented 2 years ago

Perhaps a stupid question, but how is this different from existing camera access APIs such as getUserMedia()

bialpio commented 2 years ago

It allows the sites to obtain camera images that are synchronized with WebXR's XRPoses. If a site were to obtain camera images in some other manner (like getUserMedia()), it would be unable to correlate them with spatial data that it can get by integrating with WebXR.

ekr commented 2 years ago

Does it otherwise provide the same data as getUserMedia()?

bialpio commented 2 years ago

No, there are some differences. The big one is that the camera image that is presented to the site is going to be cropped compared to the image available via getDisplayMedia(). The spec mandates that only the part of the image is aligned to the XRView will be accessible to the site, and in smartphone AR case, that cropped camera image is also displayed to the user. We also expose the projection matrix of the camera (since it's the same as the one on XRView due to the alignment requirement). The other difference is the way that the camera image is exposed - we surface the image as WebGLTexture, which the site can then use for rendering - this should be a more direct approach than obtaining ImageBitmap (via ImageCapture constructed from MediaStreamTrack) & uploading it to a WebGLTexture.