Environment

Magic Leap 2
Helio Browser
Remote Debugging via Chrome

Expected Behaviour

I expect AR Markers to render correctly where the AR Marker is.

The Problem

I am using a Magic Leap 2 headset to test WebXR development.

I am encountering an issue where AR Markers are being incorrectly rendered when I am in AR Mode after selecting the Enter AR Mode button. This is when the web browser is expanded into a full screen mode.

The issue is that when I am rendering my scene outside of AR mode, the markers are correctly translating and placed correctly in world space. It's perfect. There are no complaints.

However when I enter AR Mode my markers are incorrectly placed and offset in the opposite direction of my camera movement. I have no idea why this is.

Demo Video

I have created a video here to demonstrate the issue

https://github.com/pmndrs/react-xr/assets/1308524/bb6343c5-ad08-43db-89fb-a766fbb8dd9f

The Code

package.json

{
  "name": "@artcom/react-three-arjs-example",
  "version": "0.1.0",
  "license": "MIT",
  "scripts": {
    "watch": "webpack serve --hot --https --mode=development",
    "build": "webpack --mode=production",
    "start": "http-server -S -C cert.pem dist/"
  },
  "engines": {
    "node": ">= 16.15"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/artcom/react-three-arjs.git"
  },
  "devDependencies": {
    "@babel/cli": "^7.22.6",
    "@babel/core": "^7.22.8",
    "@babel/preset-env": "^7.22.7",
    "@babel/preset-react": "^7.22.5",
    "@pmmmwh/react-refresh-webpack-plugin": "^0.5.10",
    "babel-loader": "^9.1.3",
    "copy-webpack-plugin": "^11.0.0",
    "eslint": "^8.44.0",
    "eslint-config-prettier": "^8.8.0",
    "eslint-plugin-compat": "^4.1.4",
    "eslint-plugin-import": "^2.27.5",
    "eslint-plugin-prettier": "^4.2.1",
    "eslint-plugin-react": "^7.32.2",
    "eslint-plugin-react-hooks": "^4.6.0",
    "html-webpack-plugin": "^5.5.3",
    "http-server": "^14.1.1",
    "react-refresh": "^0.14.0",
    "webpack": "^5.88.1",
    "webpack-cli": "^5.1.4",
    "webpack-dev-server": "^4.15.1"
  },
  "dependencies": {
    "@ar-js-org/ar.js": "^3.4.5",
    "@artcom/react-three-arjs": "0.5.6",
    "@react-three/drei": "^9.88.2",
    "@react-three/fiber": "^8.14.5",
    "@react-three/xr": "^5.7.1",
    "buffer": "^6.0.3",
    "core-js": "^3.31.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "three": "^0.154.0"
  },
  "browserslist": [
    "defaults"
  ]
}

index.html

<html>
    <head>
        <title>@artcom/react-three-arjs example</title>
        <meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0" />

        <style>
            body {
            /* width: 100%;*/
            /* height: 100%;*/
            /* margin: 0px;*/
            /* padding: 0px;*/
            /* overflow: hidden;*/
            }
        </style>
    </head>

    <body>
        <div id="root"></div>
    </body>
</html>

index.js

// Third-Party
import { ARCanvas, ARMarker } from "@artcom/react-three-arjs";
import {
    Box,
    Environment,
    OrbitControls,
    // PerspectiveCamera,
    Stage
} from "@react-three/drei";
import {
    ARButton,
    // Controllers,
    Hands,
    // Interactive,
    XR,
    // useHitTest,
    useXR
} from "@react-three/xr";
import React from "react";
import { createRoot } from "react-dom/client";
// Custom
// N/A

function ConstellationStage(props) {
    // -------------------------------------------------------------------------
    // State
    // -------------------------------------------------------------------------
    const current_xr_state = useXR();

    // -------------------------------------------------------------------------
    // Rendering
    // -------------------------------------------------------------------------
    const placeholder_view = (
        <Stage
            // for formatting
            adjustCamera={true}
            intensity={1}
            preset="rembrandt"
            shadows="contact"
        >
            <Environment preset="forest" background={true} />
            <OrbitControls />
            <ambientLight />
            {/*{current_xr_state.isPresenting === false ? null : null}*/}
        </Stage>
    );
    const ar_view = (
        <>
            <Hands />
            <ambientLight />
            {/*<pointLight position={[10, 10, 0]} intensity={10.0} />*/}
            <ARMarker
                params={{ smooth: true }}
                type={"pattern"}
                patternUrl={"data/patt.hiro"}
                onMarkerFound={() => {
                    console.log("Marker Found");
                }}
            >
                <Box
                    args={[1, 1, 1]}
                    material-color="hotpink"
                    // position={[0, 1.5, -1]}
                />
            </ARMarker>
            <Box args={[0.1, 0.1, 0.1]} material-color="red" position={[0, 1.5, -1]} />
            <Box args={[0.1, 0.1, 0.1]} material-color="blue" position={[0, 1.6, -1.2]} />
            <Box args={[0.1, 0.1, 0.1]} material-color="green" position={[0, 1.7, -1.3]} />
        </>
    );
    const final_render_element = current_xr_state.isPresenting === true ? ar_view : placeholder_view;

    // return final_render_element;
    return ar_view;
}

function ConstellationScene(props) {
    const final_render_element = (
        <div
        // style={{ height: "100vh", width: "100vw" }}
        >
            <ARButton />
            <ARCanvas
                // gl={{ antialias: false, powerPreference: "default", physicallyCorrectLights: true }}
                onCameraStreamReady={() => console.log("Camera stream ready")}
                onCameraStreamError={() => console.error("Camera stream error")}
                onCreated={({ gl }) => {
                    gl.setSize(window.innerWidth, window.innerHeight);
                }}
            >
                <XR>
                    <ConstellationStage />
                </XR>
            </ARCanvas>
        </div>
    );

    return final_render_element;
}

createRoot(document.getElementById("root")).render(<ConstellationScene />);

Code Sandbox

It is available here to reproduce https://codesandbox.io/s/example-react-ar-js-vx2dj4

Speculation

I have reason to believe that there is something going on in AR Mode that is causing the rendering to get messed up with the camera itself.

I don't know why, or how.

Are there settings that I need to be mindful of when entering AR Mode?

Work Around Request

Is there a way to emulate AR Mode without using your library? I have a sneaking suspicion if I were able to do AR Mode in a different approach I might see the correct behaviour.

pmndrs / react-xr

Using `Enter AR` Button Causes Renderering Issues With Markers Resulting In Offset Positions, And AR Markers Move When Headset/Camera Changes Position #298