search

immersive-web / front-facing-camera

Incubation repository for access to front-facing-camera. Repo contact Piotr Bialecki and Rik Cabanier
Other
1 stars 1 forks source link

What if front-facing camera sessions don't support 6-DoF tracking? #2

Open tangobravo opened 1 year ago

tangobravo commented 1 year ago

I understand the rationale to separate this out from a potential "face-tracking" feature.

The explainer links to the ARCore documentation for the facing mode, but this comes along with very different behaviour in terms of tracking, plane detection, hit testing, etc. Effectively the only thing that ARCore can do in front-facing mode is face tracking, as documented here:

When the front camera is selected, ARCore's behavior changes in the following ways:

  • [...]
  • Camera.getTrackingState() will always return TrackingState.PAUSED.
  • All forms of Frame.hitTest() will always return an empty list.
  • Camera.getDisplayOrientedPose() will always return an identity pose.
  • Session.createAnchor(Pose) will always throw NotTrackingException.
  • Planes will never be detected.

Perhaps you're aware of upcoming changes to these limitations @bialpio but it seems to me like there would be no benefit for a WebXR front-facing camera mode on android vs getUserMedia. Indeed, it would inherit all of the general downsides of WebXR for smartphones that I discussed at length in https://github.com/immersive-web/webxr-ar-module/issues/77.

bialpio commented 1 year ago

Perhaps you're aware of upcoming changes to these limitations @bialpio but it seems to me like there would be no benefit for a WebXR front-facing camera mode on android vs getUserMedia.

We’d like ARCore to relax those limitations and are internally chatting about that. The perfect scenario for us (Chrome) would be to enable access to most (if not all) features associated with AR (hit-test, depth, tracking, etc) in front-facing mode, ideally on all ARCore-supported devices.

Additionally, for immersive sessions we implicitly add the “local” feature descriptor to required features collection, so on a device that does not satisfy the requirements of 6DoF tracking in front-facing camera mode we’d ideally fail the session creation request. The main challenge here may be that we (as implementers) are not going to be able to detect this at session creation time, i.e. ARCore session creation may succeed but we won’t ever get into TrackingState.TRACKING.