Support for touch/mouse click in AR Mode

[karanganesan]

Description:

• A-Frame Version: 1.0.0 / 1.0.1
• Platform / Device: Chrome Latest Android
• Reproducible Code Snippet or URL: https://glitch.com/edit/#!/hospitable-asteroid

cc @klausw He has done some work on emulating a joystick in WebXR AR Scene - https://klausw.github.io/a-frame-car-sample/index.html

[dmarcos]

One can write components listening directly to mousedown, mouseup, touchstart, `touchend‘, etc on window. A joystick is probably more appropriate as a 3rd party component since it’s very specific.

[klausw]

FYI, on a phone, immersive WebXR modes (both VR and AR) don't deliver mouse or touch events for screen touches. The immersive view just shows the content of the WebXR-specific opaque framebuffer, and the page's DOM content (including the scene canvas) is not visible and can't be touched through the phone screen while the session is in progress. On a desktop, mouse events still get processed for the page content on the 2D monitor, but that's decoupled from the 3D immersive scene.

I'm currently working on a DOM Overlay module for WebXR and a Chrome implementation which adds the ability to show a DOM element during an immersive-ar session on a phone, but that's not ready yet. See https://github.com/immersive-web/dom-overlays for current status.

[klausw]

Forgot to add, the joystick emulation in the car sample is based on WebXR ray input which uses WebXR-specific select events, that's part of the current WebXR spec and represents touches as basically XR controllers in screen space. The existing XR controller support in A-Frame kind of works for them, but there are visual glitches: https://github.com/aframevr/aframe/issues/4323

[dmarcos]

@klaus Thanks for the info. Much appreciated

[EricHsuYC]

I am sorry, is it possible to get screen touch event in AR mode in version 1.0.4 ?

[jletellier]

Currently, I'm also not able to get screen touch events in AR mode. Tried it on Chrome for Android 89. Is work on implementing the DOM overlay module for WebXR ongoing?

[SeyedAlirezaFatemi]

Is there a solution for this? I tried with the latest Chrome on Android and aframe v1.3.0 and it still doesn't work.

[diarmidmackenzie]

A-Frame now supports the dom-overlay feature of WebXR, as described here

It can be a little fiddly to set this up to enable touch event detection for Mobile AR.

This glitch example shows how this can be done.

UPDATE - see comment below for additional modifications that may be needed for iOS/Safari

Live version here

Sphere, cube & cylinder should be draggable, even in AR mode.

[diarmidmackenzie]

See also this example which shows visible DOM Overlay when in AR, but without any demonstration of touch controls:

https://aframe.io/aframe/examples/boilerplate/webxr-dom-overlay/

(this was added when DOM Overlay support was added to A-Frame: https://github.com/aframevr/aframe/pull/4509)

[ivopisarovic]

I implemented a THREE Raycaster to handle the situation. The glitch example was not working for me on iPhone in Mozilla XRViewer. There were no touch events emitted when canvas or scene was used directly as the overlay element. Hence, I had to create a transparent div to cover the canvas and provide the clickable layer.

AFRAME.registerComponent('touch-raycaster', {
    init() {
        const scene = this.el.sceneEl
        const camera = this.el.sceneEl.camera

        document.querySelector("#touch-overlay").addEventListener('click', (event) => {
            const mouse = new THREE.Vector2();
            mouse.x = (event.clientX / window.innerWidth) * 2 - 1; // convert to range (-1, 1)
            mouse.y = -(event.clientY / window.innerHeight) * 2 + 1; // convert to range (-1, 1)

            // Set the ray's origin and direction based on the camera and mouse coordinates
            const raycaster = new THREE.Raycaster();
            raycaster.setFromCamera(mouse, camera);

            // Find intersections between the ray and objects in the scene
            const intersects = raycaster.intersectObjects(scene.object3D.children);

            for (const intersect of intersects) {
                const el = intersect.object.el
                console.log('clicking on item', el)
            }
        });
    }
})

  <a-scene 
  touch-raycaster
  webxr="optionalFeatures: hit-test,dom-overlay; overlayElement: #overlay;" 
  ar-hit-test="target: #myobject"
 >
...
</a-scene>
    <div id="overlay">
      <div id="touch-overlay"></div>
    </div>

.a-dom-overlay:not(.a-no-style) {
    padding: 0 !important;
    pointer-events: auto !important;
}

#touch-overlay {
    position: fixed;
    width: 100vw;
    height: 100vh;
    top: 0;
    left: 0;
}

[dirkk0]

I put @ivopisarovic (thx!) code into a Glitch: https://glitch.com/edit/#!/dk-ar-click