Raycaster events not throwing accurate intersection points?

vishnusaivivek commented 1 year ago

First of all Mind-Ar an amazing library, but unfortunately i ran into an issue, where i was trying to use inbuilt a-frame raycaster events to detect the point of intersection on a plane, but i am not getting accurate points, i was customising the "mindar-image-target" component as per my requirement where i can add the element dynamically as soon as the target is detected and here i am attaching my code snippets.

Requirement: - Place a plane on the detected target and place a box on top of the plane for each click in the respective position.

Approach : -

Actually i want to implement multi-targets but here in this example it's just minimised to simplify the issue.

1)Here i am registering a new component called 'next-image-target' which is an exact replica of "mindar-image-target" from the library except updateWorldMatrix, as i want to add a 3D object dynamically based in target index, so i am updating the function as per my requirement and adding cursor & raycaster to the camera-entity if the target is found and removing them if target is lost.

2)Another component with the name "plane" which is binded to dynamically added plane in the updateWorldMatrix of 'next-image-target', which has raycaster dependency and i am listening to the "raycaster-intersected" & "raycaster-intersected-cleared" and storing the ray intersecting coordinates in the variable called "intersectPoints", added a click event to the plane so that i can place a box on the plane in the clicked position.

Issue Faced: - Intersected coordinates were not correct and the box were placed some where in the 3D-world.

use the below image to scan

HTML


<head>
  <script src="https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.1.5/dist/mindar-image.prod.js"></script>
    <script src="https://aframe.io/releases/1.2.0/aframe.min.js"></script>
    <script src="https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.1.5/dist/mindar-image-aframe.prod.js"></script>
  <script src="./components.js"></script>
</head>
<body>
  <a-scene mindar-image="imageTargetSrc: https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.1.5/examples/image-tracking/assets/card-example/card.mind;" vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false">
    <a-camera position="0 0 0" look-controls="enabled: false"></a-camera>
    <a-entity id="target0" next-image-target="targetIndex: 0">
    </a-entity>
  </a-scene>
</body>

components.js

let intersectPoints = new THREE.Vector3();
AFRAME.registerComponent("next-image-target", {
  dependencies: ["mindar-image-system"],
  schema: {
    targetIndex: { type: "number" },
  },

  postMatrix: null,
  init: function () {
    const arSystem = this.el.sceneEl.systems["mindar-image-system"];
    arSystem.registerAnchor(this, this.data.targetIndex);
    const root = this.el.object3D;
    root.visible = false;
    root.matrixAutoUpdate = false;
  },

  setupMarker([markerWidth, markerHeight]) {
    const position = new AFRAME.THREE.Vector3();
    const quaternion = new AFRAME.THREE.Quaternion();
    const scale = new AFRAME.THREE.Vector3();
    position.x = markerWidth / 2;
    position.y = markerWidth / 2 + (markerHeight - markerWidth) / 2;
    scale.x = markerWidth;
    scale.y = markerWidth;
    scale.z = markerWidth;
    this.postMatrix = new AFRAME.THREE.Matrix4();
    this.postMatrix.compose(position, quaternion, scale);
  },

  async updateWorldMatrix(worldMatrix) {
    const isExist = document.getElementById('plane');
    if (worldMatrix !== null && isExist===null) {
      const target = document.querySelector("#target" + this.data.targetIndex);
      let plane = document.createElement("a-plane"); 
      plane.setAttribute("id", "plane");
      plane.setAttribute("class", "plane");
      plane.setAttribute("width", "1");
      plane.setAttribute("height", "1");
      plane.setAttribute("color", "green");
      plane.setAttribute("plane", "raycast:true");
      target.appendChild(plane);
      const event = new Event("targetFound");
      document.dispatchEvent(event);
    } else if (worldMatrix === null && isExist != null) {
      document.getElementById(`plane`).remove();   
      const event = new Event("targetLost");
      document.dispatchEvent(event);
    }
    this.el.object3D.visible = worldMatrix !== null;
    if (worldMatrix === null) {
      return;
    }
    var m = new AFRAME.THREE.Matrix4();
    m.elements = worldMatrix;
    m.multiply(this.postMatrix);
    this.el.object3D.matrix = m;
  },
});

AFRAME.registerComponent("plane", {
  dependencies: ["raycaster"],
  schema: {
    raycast: { type: "boolean" },
  },
  init: function () {
    this.el.addEventListener("raycaster-intersected", (evt) => {
      this.raycaster = evt.detail.el;
    });
    this.el.addEventListener("raycaster-intersected-cleared", (evt) => {
      this.raycaster = null;
      intersectPoints = null;
    });
    this.el.addEventListener("click", (evt) => {
      console.log(intersectPoints);
      if (intersectPoints) {
        let box = document.createElement("a-box");
        console.log(`${intersectPoints.x} ${intersectPoints.y} ${intersectPoints.z}`);
        box.setAttribute("position", `${intersectPoints.x} ${intersectPoints.y} ${intersectPoints.z}`);
        box.setAttribute("material", "color:red");
        box.setAttribute('depth','10');
        box.setAttribute('height','10');
        box.setAttribute('width','10');
       this.el.appendChild(box);
      }
    });
  },
  tick: function () {
    if (!this.raycaster) {
      return;
    } 
    let intersection = this.raycaster.components.raycaster.getIntersection(this.el);
    if (!intersection) {
      return;
    } 
    intersectPoints = intersection.point;
  },
});

document.addEventListener("targetFound", () => {
document.querySelector('a-camera').setAttribute('raycaster','far: 10000;objects: .plane');
document.querySelector('a-camera').setAttribute('cursor','fuse: false; rayOrigin: mouse;');
});

document.addEventListener("targetLost", () => {
  document.querySelector('a-camera').removeAttribute('raycaster')
  document.querySelector('a-camera').removeAttribute('cursor')
});

Finally another point of doubt is that the position of the model shows 0,0,0 always from the a-frame inspector, is there any way which reveals the model placing position, as per my observation its completely handled by the world matrix, Any idea how to convert world matrix in to world coordinates.

Thanks in advance for the help.

vishnusaivivek commented 1 year ago

@hiukim @Makio64 @Parikshit-Hooda can anyone please look into this issue, any help would be appreciated.

sudharsans85 commented 1 year ago

Facing same issue with respect to raycasting. May be we are missing something. Any help will be appreciated.

hiukim commented 1 year ago

fixed in v1.2.2

hiukim / mind-ar-js

Raycaster events not throwing accurate intersection points? #266