Interactive Regions API

[mkeblx]

An Interactive Regions API would give the ability to define interactive regions within an immersive WebXR experience, allowing the OS (or the browser) to highlight these regions when gazed upon, providing users with intuitive visual feedback in a privacy conscious way.

The WebXR experience would be responsible for creating interactive regions and attaching them to an active WebXR session, and optionally listening for 'click' events on these regions.

Background

Hover effects are useful and ubiquitous on the web but they raise serious privacy concerns if they are driven directly by a user's gaze (e.g. via eye tracking on an HMD) in a naive way. To strike a balance between usability and user privacy, Apple introduced a privacy-preserving method with its visonOS wherein elements are visually highlighted when a user looks at them, but crucially the highlighting is done only by the OS, without any applications having knowledge of the highlighting state.

Applications must be able to specify the areas to highlight, 'interactive regions', to the OS. In the case of the Safari browser, the WebKit engine generates these regions on the 2D page from various pieces of webpage markup and styles. The webpage thus determines where these interactive regions are.

Extending this concept into the 3D world of WebXR would be a logical progression. While WebXR doesn't rely on traditional 2D DOM elements which a web engine could automatically identify and highlight, immersive experiences have their own set of 3D elements that users can interact with that would greatly benefit from being able see highlighted like they can see elsewhere in their interactions on the platform. As the browser can't automatically identify regions, the web developer would have to explicitly supply them.

References:

WWDC Safari talk, discussion of 2D web interactive regions: https://developer.apple.com/videos/play/wwdc2023/10279/?time=288

Code:

An example of what usage of Interactive Regions would look like:

// Engine object
const position = { x: 1.0, y: 1.5, z: -3.0 };
const orientation = { x: 0, y: 0, z: 0, w: 1 };
Const scale = { x: 0.5, y: 0.2, z: 0.1 };
Let floatingButtonObj = new Engine.Button3D('floatingButton', position, orientation, scale);

// create an Interactive Region
let interactiveButtonRegion = new XRInteractiveRegion();
interactiveButtonRegion.regionId = 'floatingButton';
interactiveButtonRegion.transform = new XRRigidTransform(position, orientation); 
interactiveButtonRegion.width = 0.5; // meters
interactiveButtonRegion.height = 0.2;
interactiveButtonRegion.style = new XRRegionAppearanceAttributes({ highlightRadius: 0.1 });

xrSession.interactiveRegions.setSpace(xrSpace);

// Add region
xrSession.interactiveRegions.addRegion(interactiveButtonRegion);

// Handle events
//  new event
xrSession.addEventListener('interactiveregionclick', (event) => {
    if (event.regionId === 'floatingButton') {
        // Handle interaction
        floatingButtonObj.click();
    }
});

// Or could use 'select'
xrSession.addEventListener('select', (event) => {
  let source = event.inputSource;
  let frame = event.frame;
  let targetRayPose = frame.getPose(source.targetRaySpace, myRefSpace);

  // If the interaction source has an associated region
  if (source.interactionRegion) {
    if (source.interactionRegion.id === 'floatingButton') {
        // Handle interaction...
    }
  } else {
    let targetObject = findTargetUsingRay(targetRay.transform.matrix);
    // ...
  }
});

Prototype:

.

[mkeblx]

@AdaRoseCannon @toji I just noticed today this PR & thread with lots of good related discussion I'm reading through now and will comment on, but I also have the above for trying to provide a method of getting 'hover' feedback.

I will share more fleshing out but would appreciate if have any initial feedback or thoughts. Thanks!

[AdaRoseCannon]

Thank you so much for putting this together. It is something I have been thinking about for a while, I was thinking something based around the layer API could be a good fit. I really like that they can be selected which is something I think would be really important.

[AdaRoseCannon]

I'd probably do something like if (event.target === interactiveButtonRegion) instead of relying on an id.

[mkeblx]

Yes a similar functionality could be accomplished with the Layer API as well, say with XRQuadLayers marked interactive etc.

It would be good to establish more broadly the need for an API, of whatever exact form, that accomplishes 'hover' support. Probably a good call discussion topic.

Is there a solution besides developer explicitly supplying a list? A (stereo) segmentation mask texture could work, but could be overly complicated.

What level of visual hover is sufficient? The above supports rectangular (and rounded rectangle / circle) regions which wouldn't cover every scenario a developer might want hover highlight to support (i.e. highlight of a character model) but could be sufficient for many scenarios.

[cabanier]

Your proposal is very close to the one that was made for canvas hit regions. However, in this case, it might make more sense to draw pixels instead of paths because that is easier to do for a WebGL experience.

The important part of that proposal is that there is access to a "fallback DOM" which will give the screen reader access to the accessibility tree.

[kfarr]

I encountered this issue today during on-site device testing and filed a WebKit Accessibility issue with Apple before even seeing this thread.

Here is a link to the issue: https://feedbackassistant.apple.com/feedback/13464533

Pasting below for convenience since the link above requires a login:

Please provide a descriptive title for your feedback: Enhancement Request: How to define gaze hover zones in 3D scene with WebXR? Example: Core Data should support nil values as defaults

Which platform is most relevant for your report? visionOS

Which technology does your report involve? WebKit

What type of feedback are you reporting? Suggestion

What does the WebKit issue you are reporting involve? Accessibility

Summary:

• As a WebXR developer creating immersive scenes, I wish to empower the user to see visual "hover zones" using eye tracking that are positioned as elements in the 3D scene defined by three.js / WebXR API standard (instead of standard 2D DOM Elements).

More specifics:

• We have made a "tile builder" app for WebXR that is used in both immersive and VR mode on other headsets with controllers. As a user on these other headsets they can "hover" with a raycaster showing highlighted "cells" in a grid prior to placing an object.
• To port to Vision Pro and reflect the same UI standard in visionOS, our design goal is to use eye gaze as the "hover" motion and allow the user to see the grid of tiles as a collection of 2D planes, one for each cell of the grid, that respond to gaze hover events so that a user has confirmation that when they do a "pinch" event that the selected object will place the object in the grid cell as expected.

Issue:

• There does not appear to be a mechanism in WebXR to provide gaze hover hints in a WebXR immersive environment. This issue affects menu interfaces in immersive mode as well as spatial object placement design interfaces.

Workaround:

• App design change: Use a "live updated" raycaster controlled by hand motion, instead of eye gaze, to preview where the object will be placed.
• Use 2D DOM, hacky, probably won't work: Use 3D CSS to align placement of a grid of divs with appropriate css hover info aligned on top of the three.js canvas.

Example, still intended for controllers, not retrofit for hand control:

• Simplified URL (intended for controllers, not hands): Code: https://glitch.com/~aframe-2-vr-menus Live Preview: https://aframe-2-vr-menus.glitch.me/
• More advanced project (intended for controllers, not hands): Code: https://github.com/3DStreet/citybuilderjr Live Preview: https://3dstreet.github.io/citybuilderjr/