search

immersive-web / webxr

Repository for the WebXR Device API Specification.
https://immersive-web.github.io/webxr/
Other
2.99k stars 384 forks source link

Allow WebXR pages to advertise their Focus Awareness, add blurred visibility state #1008

Closed kearwood closed 4 years ago

kearwood commented 4 years ago

There are some use cases where the browser or OS may wish to display system UI on top of the WebXR rendered content without interrupting the site.

In this case, the headposes would continue to be updated with the RAF loop; however, the page would be expected to stop rendering controllers and would not be able to retrieve poses for the controllers while the page is in the "blurred" state.

This "blurred" state could be added to the XRVisibilityState enum. The page could perhaps advertise its awareness of Focus by passing a flag to the options argument of navigator.xr.requestSession.

kearwood commented 4 years ago

As an example for such focus awareness at a platform / OS level:

https://developer.oculus.com/documentation/native/android/mobile-overlays/

kearwood commented 4 years ago

Such "blurred" state may be used to display information (eg, media transport controls) that are not security sensitive. The UA will ensure that any displayed UX during such a blurred state is designed in such a way that no security sensitive information from the headpose can be inferred. (eg, a permission prompt asking for microphone access would randomize the order of the "Accept" and "Reject" buttons).

cabanier commented 4 years ago

Doesn't the spec already offer this with the "visibility-blurred" event? Oculus is using this to display system dialogs with the immersive session still drawing in the background without controllers.

kearwood commented 4 years ago

It seems that this could be implemented as a refinement of visibility-blurred, with some additions:

Alternately, we could require pages to handle visibility-blurred always and reject the XRSession.getPose promise when an XRInputSource's XRSpace is passed in.

kearwood commented 4 years ago

Something else to consider.. If we ask WebXR pages not to show controllers when visibility-blurred, this may need to be more clearly defined...

Some pages may not actually show a "controller" that looks like an object held in the hand. It could, for example, drive an IK rig with fingers, a hand, and elbow. In this case, it would look odd to see the avatar's arm disappear. Rather, the text should say something about tracking movement or representing a "pointer"...

kearwood commented 4 years ago

Perhaps if a page could advertise that it supports visibility-blurred in the session creation options, we could then activate otherwise compatibility breaking behavior, such as rejecting the XRSession.getPose promise when used to get the pose of a controller during visibility-blurred.

cabanier commented 4 years ago

Perhaps if a page could advertise that it supports visibility-blurred in the session creation options, we could then activate otherwise compatibility breaking behavior, such as rejecting the XRSession.getPose promise when used to get the pose of a controller during visibility-blurred.

Have you seen experiences that break? So far, I've only seen experiences that show the controllers as frozen instead of not rendering them in the visibility-blurred state .

kearwood commented 4 years ago

Have you seen experiences that break? So far, I've only seen experiences that show the controllers as frozen instead of not rendering them in the visibility-blurred state .

No sites that break; although, it would be perhaps a recommendation for sites to hide the controllers so it doesn't look broken or confuse users that may see two sets of controllers.

Perhaps we could resolve this by clarifying the "Input is not processed by the XRSession.". One way would be to just say that the XRPose returned by getPose for a XRInputSource would continue to return the same transform for the duration of the visibility-blurred state, but still resolve the promise successfully. (IIUC, this is the Oculus browser behavior currently?)

We could then add non-normative text informing sites that the best practice is to present the controllers in a way that indicates that they are no longer tracked. (eg, hide anything that looks like the physical controller, or switch to default animations or pose for rigged avatars...) When the visibility-blurred state ends, the site should be aware that there may be a large jump of movement for the controllers as they resume tracking (and to avoid it affecting physics, gameplay, etc...).

kearwood commented 4 years ago

Should we agenda this? If others agree, I would like to make a PR to just update the spec text to be more explicit on what "Input is not processed by the XRSession." does. The result would effectively be the same as what the Oculus browser does currently, with non-normative recommendations on hiding controller models that may be confused with the UA's rendered models.

cabanier commented 4 years ago

You don't need to put this on the agenda since I'm pretty sure that we're all in agreement. If you can add recommendations to the spec and the explainer, I'm sure that will be enough.

Maybe we should also file bugs against three.js and other frameworks so they add listeners and behavior for these events.

kearwood commented 4 years ago

This sounds great to me. I'll prepare a PR on Monday with the proposed text changes.

I've talked about this internally at Mozilla and it seems that it can work for us. If we really do need pages to advertise focus awareness, that can always be done additively later on. I hope that it wouldn't be necessary; however.

cabanier commented 4 years ago

Perhaps we could resolve this by clarifying the "Input is not processed by the XRSession.". One way would be to just say that the XRPose returned by getPose for a XRInputSource would continue to return the same transform for the duration of the visibility-blurred state, but still resolve the promise successfully. (IIUC, this is the Oculus browser behavior currently?)

Yes, that is what the Oculus browser does. I'm pretty certain that other Chromium based implementations would do the same.

Manishearth commented 4 years ago

getPose() isn't a promise, but yes, in this case getPose() is supposed to return null.

Manishearth commented 4 years ago

  • While blurred, the page is expected not to display controllers. (The spec currently only says, "Input is not processed by the XRSession.")

  • XRInputSource XRPose's will not be updated while in visibility-blurred.

I believe this is the intent of the current spec text. When input is not processed; input poses are not extant.

  • WebXR sessions would have an options flag or feature to specify that they will respond to the "visibility-blurred" state. (The UA may choose a different UX if a page does not).

Not sure if this is strictly necessary? If the input source pose is null, there's not much you can do other than stop rendering the controller.

You can already trigger this case by sticking your controller out of tracking range. I would definitely expect applications to handle this properly, especially as the failure mode of this is that you get a null error.

I don't think there needs to be special handling for the blurred state aside from the app taking it as a hint to pause things if necessary.

I would suggest we fix this by expanding the note about input not being processed.

cabanier commented 4 years ago

I don't think there needs to be special handling for the blurred state aside from the app taking it as a hint to pause things if necessary.

Yes, things like the game logic and video playback need to be stopped. In the case of the Oculus browser, we also throttle the WebXR session so the UI overlay is responsive.

Manishearth commented 4 years ago

Yes, things like the game logic and video playback need to be stopped.

Right, apps can choose to handle visible-blurred if they want to, but there's no reason to force them to handle it, or have some kind of negotiation protocol so the UA can react to the app's ability to handle it.

The initial need for such a negotiation was so the UA would know what to do about rendered controller models -- but if null poses solve this then we don't need negotiation.

Manishearth commented 4 years ago

I've extracted the core issue out of here and into https://github.com/immersive-web/webxr/issues/1042

Let me know if there are still things that need to be determined here.