Camera resolution setting

[whatisor]

Currently no api found for setting camera resolution in AR session.

[cperey]

In early 2013, the Khronos Group established a working group to develop a standard for advanced control of camera and other sensors. See the page for the use cases and requirements. For AR, they included:

• 60Hz, low-latency capture with motion sensor synchronization
• Multiple Region of Interest (ROI) capture
• Multiple sensors for scene scaling
• Detailed feedback on camera operation per frame

If the work had continued, this standard would have provided the API you are seeking.

After evaluating the suitability of FCam (an open source project developed in coordination between Nokia Research under leadership of Kari Pulli, and Stanford Computer Graphics Lab), there was insufficient member engagement and, unfortunately, the OpenKCam work froze before reaching its goal.

I hope to hear if such an activity continued/was completed elsewhere (e.g., in another standards development organization), or if there is a defacto standard today that meets the needs outlined in the OpenKCam WG.

If there are people who reply to this comment that they would have use for a general API for camera and sensor control for AR and there is not an alternative in use, the OpenKCam WG can resume its work towards such a goal.

Looking forward to replies and comments. Happy New Year!

[toji]

@whatisor: Could you expand a bit more on what you're looking to accomplish? "setting camera resolution" could potentially be read a couple of ways.

If you are looking for ways to reduce or increase the resolution of rendered content you'll want to look at the existing framebufferScaleFactor layer creation argument and the requestViewportScale() method.

If you're looking more for how to query or control the camera intrinsics, we haven't considered that yet (aside from providing the appropriate projection matrices for rendering) and I would like to know a little bit more about your intended use cases to get a better sense of what's missing.

[whatisor]

Hi @toji When access the AR session, camera resolution is always defaulted to small resolution, so image quality is low.

If you're looking more for how to query or control the camera intrinsics

==> Yes, I want to control and query the camera parameters as the normal android camera.

Thank you.

[toji]

That's unexpected. Unless the app is using one of the above APIs to intentionally lower the image quality the defaults should use effectively the full resolution. Would you mind letting us know what device you are testing with and what pages you're seeing the issue with?

[klausw]

While I think it's currently working as expected, @whatisor is correct in the sense that phone cameras often have a higher resolution than the screen does. For example, a Pixel 5 has a 3024x4032 pixel camera and 1080x2340 screen. Currently, the phone AR implementation on Chrome runs at screen resolution by default, and this is lower than the available camera resolution. The camera image is also cropped to fill the screen since the camera sensor typically has a different aspect ratio.

In general implementations should have the flexibility to pick appropriate defaults where this isn't the case, for example using a lower default framebufferScaleFactor for drawn content on a device with a high-resolution screen but comparatively underpowered GPU. As far as I know this isn't currently being used, the default framebufferScaleFactor for phone AR on Chrome uses full 1:1 screen resolution, and I think that ARCore uses an at least screen-sized camera image. If that's not the case for you, can you file a bug at https://crbug.com to follow up?

Note that the current WebXR AR API doesn't provide access to the raw camera image, and the image is only visible on the phone screen, so there wouldn't be any benefit to using a higher resolution camera image.

In the future, I think there would potentially be a benefit to providing a full resolution camera image as part of the "raw image access" API, but it's unclear if this is feasible for current implementations. See https://github.com/bialpio/webxr-raw-camera-access/blob/master/explainer.md , I'd suggest filing an issue there if that's something you'd be interested in.

Bigger picture, I do think there would be value in providing WebXR APIs to provide hints or requests related to expected camera usage. Specifically, ARCore offers a choice between fixed focus and autofocus with different tradeoffs, and there's no way to express this as part of session creation.

[Maksims]

Specifically, ARCore offers a choice between fixed focus and autofocus with different tradeoffs, and there's no way to express this as part of session creation.

I've just recently experienced a need of focus change, and was a bit surprised, that my AR prototype simply was not good (blurred camera image within 10-50cm distance) as it was mainly meant for close up objects.

There are definitely cases with AR applications that are focused (no pun intended) on close and small things, like mini figures, tabletops, on table experiences, etc.