Hit-test from arbitrary screen location (without user input)?

[DsRQuicke]

Does the hit-test API currently support generating a ray at an arbitrary screen location, without explicit user interaction, on a touchscreen device (i.e. transient input source)?

The reason I'm asking, is because I was thinking about experimenting with an app where an image/object would be detected in the camera view. And then perform a hit-test from the location of the object on the screen, to get the pose in 3D space. So this would be somewhat similar to augmented images of ARCore, or the Experimenting with computer-vision in WebXR proof-of-concept.

This use case would probably be covered by the computer-vision feature proposal. But I assume that it isn't getting out of incubation in the near future.

Maybe I should somehow just get the user to "select" the object/image on the screen. Or is there another recommended approach I could take?

[bialpio]

Does the hit-test API currently support generating a ray at an arbitrary screen location, without explicit user interaction, on a touchscreen device (i.e. transient input source)?

Yes - based on what you described, I think you'll have to calculate the ray corresponding to a specific screen location where the tracked object is, convert the ray to be expressed relative to local reference space, and subscribe to hit test with the calculated ray and local reference space. This assumes that the object you're detecting is static and that you have a programmatic way of knowing where the object is (currently, we do not expose the camera image through WebXR).

Maybe I should somehow just get the user to "select" the object/image on the screen. Or is there another recommended approach I could take?

This approach would probably be the easiest to implement now as it does not rely on features other than hit test. You can either subscribe to transient input source and ask the users to touch the object you're interested in, or subscribe to input source with viewer reference space and ray facing forward while displaying the reticle and tell the user to touch the screen when the reticle is in the spot they want (i.e. on the object). This page might give you some idea on how to use the API in this way.

[DsRQuicke]

Thank you for your response. If I understand your first suggestion correctly then it would look somewhat like this:

let localHitTestSource = null;

function onObjectFound() {
    xrSession.requestHitTestSource({
        space: localSpace, // XRSession.requestReferenceSpace("local")
        offsetRay: <ray_from_viewer_offset_to_local>
    }).then((hitTestSource) => {
        localHitTestSource = hitTestSource;
    });
}

function renderFrame(ts, xrFrame) {
    // Regurlar scene & camera updates ...

    if ( localHitTestSource ) {
        let hitTestResults = xrFrame.getHitTestResult(localHitTestSource);
        if ( hitTestResults.length > 0 ) {
            // Get the detected objects location from the hit test result ...
        }
        localHitTestSource.cancel();
        localHitTestSource = null;
    }
}

[bialpio]

Yes, that's how I'd attempt to implement it. Few minor comments:

• I'm reading the <ray_from_viewer_offset_to_local> as if it means "a ray from viewer to the interesting object, expressed in local space".
• I think it might be better to cancel() & null the localHitTestSource only after the localHitTestSource returned some results for the first time (i.e. move that inside the if(hitTestResults.length > 0) block) - otherwise, it might turn out that even though the hit test source was created successfully, it doesn't have the results available yet and will not live long enough to actually return anything.

[DsRQuicke]

Hi, thanks again for your feedback.

"a ray from viewer to the interesting object, expressed in local space"

That is indeed what I meant.

And good point about moving the cancel() inside the if-test. Additionally I should then probably also add some sort of timeout. Otherwise the XRHitTestSource might linger around forever, if it wouldn't hit anything at all.