search

jeromeetienne / AR.js

Efficient Augmented Reality for the Web - 60fps on mobile!
MIT License
15.79k stars 2.22k forks source link

Conflict with... cursor="rayOrigin: mouse" #148

Closed dreinstein10 closed 5 years ago

dreinstein10 commented 7 years ago

First of all, congratulations for your great work, it's awesome.

The latest a-frame version 0.6.1 now includes Mayognaise "aframe-mouse-cursor-component" which works fine on its own. I'm trying to integrate this component into aframe-ar by adding: cursor="rayOrigin: mouse" on the a-marker-camera but I get this error:

three.js:15179 Uncaught TypeError: Cannot read property 'count' of undefined at Lt.Object.assign.raycast (three.js:15179) at Xn (three.js:39252) at jn.intersectObjects (three.js:39329) at i.tick (raycaster.js:167) at a-scene.js:517 at Array.forEach () at HTMLElement.value (a-scene.js:515) at HTMLElement.value (a-scene.js:562) at HTMLElement.render (bind.js:12) at e (a-scene.js:459)

This is the code I'm using:

<html>
  <head>
    <script src="https://aframe.io/releases/0.6.1/aframe.min.js"></script>
    <script src="https://rawgit.com/jeromeetienne/ar.js/master/aframe/build/aframe-ar.js"></script>
    <script>
      THREEx.ArToolkitContext.baseURL = 'https://rawgit.com/jeromeetienne/ar.js/master/three.js/'
    </script>
  </head>
  <body>
    <a-scene  embedded artoolkit='sourceType: webcam;'>
      <a-box position="0 0 0" rotation="30 30 0" color="skyblue"
             event-set__enter="_event: mouseenter; material.color: yellowgreen; scale: 3 1 1"
             event-set__leave="_event: mouseleave; material.color: skyblue; scale: 1 1 1">
        <a-animation attribute="rotation" begin="click" dur="500" fill="backwards" to="30 30 360"></a-animation>
      </a-box>
      <a-marker-camera  preset='hiro'  cursor="rayOrigin: mouse"> </a-marker-camera>
    </a-scene>
  </body>
</html>

Everything works fine until I add... cursor="rayOrigin: mouse". I guess this is not currently supported because it's a new aframe component, but I was wondering if anyone else has tried to use it with success. And if not, where can I start digging into to try to fix this issue?

Thanks in advance Keep up the awesomeness !

JonathanWade commented 7 years ago

I cam to post the exact same issue so +1 on this. Happens both on desktop chrome and iOS11 Safari.

jeromeetienne commented 7 years ago

here this is due to the camera having a custom projection matrix. and this being trouble with three.js. aka this is a complex hairy problem to fix.

that said here people just want to be able to click on object. and this one could be provided more easily! let me reread this issue under this angle :)

jeromeetienne commented 7 years ago

@dreinstein10 sorry i dunno the cursor= stuff. can you provide me a simple example of it working in webvr. Just for me to quickly learn more about this component. i have the feeling it will be faster for resolving this issue

update: no more need. i did my homework see below. thanks :)

jeromeetienne commented 7 years ago

from here - https://mayognaise.github.io/aframe-mouse-cursor-component/basic/index.html

<html>
  <head>
    <script src="../build.js"></script>
    <link rel="stylesheet" type="text/css" href="../common.css"/>
  </head>
  <body>
    <a-scene>
      <a-box position="0 3.5 -2" rotation="30 30 0" color="skyblue"
             event-set__enter="_event: mouseenter; material.color: yellowgreen; scale: 3 1 1"
             event-set__leave="_event: mouseleave; material.color: skyblue; scale: 1 1 1">
        <a-animation attribute="rotation" begin="click" dur="500" fill="backwards" to="30 30 360"></a-animation>
      </a-box>

      <a-sky color="pink"></a-sky>

      <a-entity position="0 1.8 4">
        <a-entity camera look-controls mouse-cursor>
          <a-cursor fuse="true" color="yellow"></a-cursor>
        </a-entity>
      </a-entity>

      </a-entity>
    </a-scene>
    <div class="spacer"></div>
    <div class="spacer2"></div>
  </body>
</html>
jeromeetienne commented 7 years ago

ok so i got a lead...

Currently aframe-mouse-cursor-component use the rendering camera to do the raycasting. This is perfect in a normal case. But here it trigger an issue in three.js raycasting and how it interfere with the AR camera calibration. So we need to work around that. and this can be done if aframe-mouse-cursor-component can use the camera we would specify and not the rendering camera.

hello @mayognaise, are you following all this ? :)

jeromeetienne commented 7 years ago

at first sight, it seems around here

https://github.com/mayognaise/aframe-mouse-cursor-component/blob/master/index.js#L396

JonathanWade commented 7 years ago

@jeromeetienne This has slightly changed (hence that cursor= part) as A-Frame 0.6.1 from 15 days ago has rolled the functionality in with a similar, but not duplicate implementation to @mayognaise 's component 0.6.1 Release notes

See this pull request

dreinstein10 commented 7 years ago

Hi @dmarcos, do you know about this ?

db2crush commented 7 years ago

Hi, I also got this error..
Is this problem solved?

kareraisu commented 7 years ago

Also got bit by this. Dear @jeromeetienne, could you make any progress thus far?

JamesBotterill commented 7 years ago

This is afflicting me also

sameekhan commented 7 years ago

Having issues with this as well. Got to the end of my MVP and realized I have to choose between click functionality on mobile and the camera lol. @jeromeetienne let me know if I can help in anyway.

javismiles commented 6 years ago

any progress here?, need to be able to detect a click on an element urgently, but so far clicks are detected across the entire canvas and screen, how could we detect a click on an specific element triggered by the marker?

mrturck commented 6 years ago

I was able to workaround this issue by using event-set-component in addition to mouse-cursor-component. Was able to set visual properties, although no luck with running functions from clicks.

Ex using: a-frame 0.7.1, event-set-component (4.0.0) mouse-cursor-component a-frame-ar

I use javascript to create a box with the mouse click functionality

var box = document.createElement('a-box'); plane.setAttribute('event-set__down','_event: mousedown; scale: 2 2 2') scene.appendChild(box);

perrychen901105 commented 6 years ago

Hello, I am facing the same problem, I want to click on the object, but it is not work. Here are my code. `

  </a-marker>
  <a-entity position="0 0 0">
    <a-entity camera look-controls mouse-cursor>
      <a-cursor id="cursor" fuse="true" color="yellow" raycaster="objects: .link">
      </a-cursor>
    </a-entity>
  </a-entity>`

and the click js is, `AFRAME.registerComponent('changescene', { schema: { event: {type: 'string', default: ''}, linkto: {type: 'string', default: ''} },

multiple: true,

init: function () { var self = this; },

update: function () { var data = this.data; var el = this.el; if (data.event) { console.log("googd"); el.addEventListener(data.event, function () { console.log("click successful"); alert("clicked"); }); } } });` Thanks.

Yggdragstyle commented 6 years ago

Hello,

I search an solution for this problem, and I have try to use a Three.js raytracer for get a clickable element. But I' noob, so I grope and I fail...

this is my function : (in orign I code in Coffee Script, but I translate for you in es6 (more and less))

AFRAME.registerComponent( 'clickable-ar', {

init: () => {

        document.addEventListener('click', () => {

            raycaster = new THREE.Raycaster()
            mouse = new THREE.Vector2()

            mouse.x = ( event.clientX / window.innerWidth ) * 2 - 1
            mouse.y = - ( event.clientY / window.innerHeight ) * 2 + 1

            # raycaster.setFromCamera( mouse.clone(), AFRAME.scenes[0].camera )
            # objects = raycaster.intersectObjects( AFRAME.scenes[0].sceneEl.children )

            # position = {x: 0, y: 0, z: 0} and  direction = {x: 0, y: 0, z: -1} )
            raycaster.set( AFRAME.scenes[0].camera.position, AFRAME.scenes[0].camera.getWorldDirection() )
            elts = raycaster.intersectObject( this, true )

            for( elt in elts ) {
                elt.click(); 
            }
    });
}

});

I' want to know if I'm in a good way, or it's ridiculus ? And any suggestion..

Sorry for my very bad English.. I'm French :P And thank you for your contribution ! Ygg.

enquel commented 6 years ago

Hi guys. After a real nightmare session I've managed to get AFrame fuse cursor working with marker-camera. It stopped disappearing on marker found.

It's kind of hack/exploit, but works really well. Below only the parts that matter (please note there is no hyphen in a-marker-camera):

<a-marker camera preset='hiro'> <a-plane id="buttRegion1" scale="1 1 1" position="0 0 0" opacity="1" rotation="-90 0 0" width="1" height="1" color="#7BC8A4">
   </a-plane> </a-marker camera> 

 <a-entity camera></a-entity>
     <a-entity cursor="fuse: true;"
                          position="0 0 -1"
                          scale="0.02 0.02 0.02"
                          geometry="primitive: ring"
                          material="color: black; shader: flat">
                    <a-animation begin="click" easing="ease-in" attribute="scale" dur="150"
                                 fill="forwards" from="0.02 0.02 0.02" to="0.03 0.03 0.3"></a-animation>
                    <a-animation begin="cursor-fusing" easing="ease-in" attribute="scale" dur="1500"
                                 fill="backwards" from="0.03 0.03 0.03" to="0.02 0.02 0.02"></a-animation>
                </a-entity>

and for the event, in my case, this:

<script>
      document.querySelector('#buttRegion1').addEventListener("click", function(e)
      {
         console.log("buttRegion1");
          window.location.href = './Video1/index.html';
          this.style.display = 'none';
      }, false);
    </script>
nicolocarpignoli commented 6 years ago

i'm closing this, as we got a working solutions. also topic seems obsolete.

Steakeye commented 6 years ago

@nicolocarpignoli: If @enquel's workaround/hack was working before, it isn't now! image

nicolocarpignoli commented 5 years ago

This PR has been closed due to inactivity. Please open a new, updated issue if the problem or the question is still relevant.

taime commented 4 years ago

Everything became extremely slow, when adding rayOrigin: mouse (I need to handle click event on 3D-model - that's why I use rayOrigin: mouse)

AndrewChildersClark commented 4 years ago

so it looks like no one has really figured this out still. I got it working in react web ar, but does not work well. Triggering animations by clicking entities has some issues with recognizing the click. the raycaster is not fine tuned enough. Please I hope someone finally figures this out, or updates ar.js. I wish I could be of more help. I will post my react example when I have some time to do a write up.

kvabakoma commented 4 years ago

I got it working with a hacky solution. It is far from perfect but test users didn't notice anything.

The problem As far as I understand the problem is that arjs uses a different projection matrix than the actual rendering camera, so when you initiate a mouse raycast it is not aimed properly as it uses a camera with different properties. None of the solutions I found online seemed to work on a mobile.

The ugly solution My logic was that a raycast originating from the cursor (center) should aim at the same direction no matter the camera projection matrix. So when a user clicks anywhere on the screen we can send a raycast from the cursor and then calculate the offset using the collision point, distance and angle.

  1. Create a large plane as a child of the <a-marker and give it the .collidable class
  2. When the user clicasdks on the screen send a raycast from the cursor
  3. Detect the collision point on the plane
  4. Calculate the actual desired collision position (get where the user clicked on the screen, multiply it by a distance factor and place it on the plane accordingly - this is your raycast hit)
  5. Cycle through your real collidable objects and check if the distance between the new hitpoint and the the collidbale is bellow your threshold.

HTML:

<a-scene embedded renderer=" colorManagement: true;  logarithmicDepthBuffer: true" arjs="debugUIEnabled: false;  detectionMode: mono_and_matrix; patternRatio: 0.5" vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false">
      <a-marker id="marker" type="barcode" value="5" smooth="true" marker-detected keep-model-visible>
        <a-plane raycaster-listen class="collidable" color="#CCC" height="20" width="20" rotation="-90 0 0" material="opacity:0.001"></a-plane>
      </a-marker>

      <a-box scale=".1 .1 .1" id="debug-box"></a-box>
      <a-entity camera>
        <a-entity id="ray-rot-x">
          <a-entity raycaster="objects: .collidable; showLine: false; far: 30; useWorldCoordinates: true" id="ray-rot-y"></a-entity>
        </a-entity>
        <a-entity cursor="fuse: false" position="0 0 -1" geometry="primitive: ring; radiusInner: 0.02; radiusOuter: 0.03" material="color: white; shader: flat; opacity:.05"> </a-entity>
      </a-entity>
    </a-scene>

JS:

AFRAME.registerComponent("raycaster-listen", {
  init: function () {
    // Use events to figure out what raycaster is listening so we don't have to
    // hardcode the raycaster.
    let el = this.el;
    let myRaycaster;

    let canvas = this.el.sceneEl.canvas;
    let clickPosUpdated = new THREE.Vector3();
    let distanceMultiplierX = 0.6;
    let distanceMultiplierY = 0.45;
    let isMobile = this.el.sceneEl.isMobile;

    this.el.addEventListener("raycaster-intersected", (evt) => {
      this.raycaster = evt.detail.el;
      myRaycaster = this.raycaster;
    });
    this.el.addEventListener("raycaster-intersected-cleared", (evt) => {
      this.raycaster = null;
      myRaycaster = null;
    });

    canvas.addEventListener("touchstart", function (evt) {
      console.log(evt);
      // GET MOUSE XY
      var _canvasSize = canvas.getBoundingClientRect(),
        w = _canvasSize.width,
        h = _canvasSize.height,
        offsetW = _canvasSize.left,
        offsetH = _canvasSize.top;
      var cx = void 0,
        cy = void 0;
      if (isMobile) {
        var touches = evt.touches;
        if (!touches || touches.length !== 1) {
          return;
        }
        var touch = touches[0];
        cx = touch.clientX;
        cy = touch.clientY;
      } else {
        cx = evt.clientX;
        cy = evt.clientY;
      }
      cx = cx - offsetW;
      cy = cy - offsetH;
      var mouseX = (cx / w) * 2 - 1;
      var mouseY = -(cy / h) * 2 + 1;

      // RAYCAST
      let intersection = myRaycaster.components.raycaster.getIntersection(el);
      if (!intersection) {
        return;
      }
      // console.log(intersection.point);

      // CALCULATE UDPATED POS
      clickPosUpdated.x = intersection.point.x + intersection.distance * distanceMultiplierX * mouseX;
      clickPosUpdated.y = intersection.point.y + intersection.distance * distanceMultiplierY * mouseY;
      clickPosUpdated.z = intersection.point.z;
      console.log("clickPosUpdated", clickPosUpdated);

      // SET DEBUG BOX
      debugBox.setAttribute("position", clickPosUpdated);
    });
  },
});

Left to do: calculate the Z offset based on the angle of the collision / rotation of the marker.

EDIT: fixed the code. it seemed I had uploaded a wrong version.

dmarcos commented 4 years ago

What’s the difference about the ar.js camera? Is not a perspective one with a pose? Has the cursor ever worked? When did it break?

Diego

On Tue, Mar 31, 2020 at 1:12 AM Nikola Georgiev notifications@github.com wrote:

I got it working with a hacky solution. It is far from perfect but test users didn't notice anything.

The problem As far as I understand the problem is that arjs uses a different projection matrix than the actual rendering camera, so when you initiate a mouse raycast it is not aimed properly as it uses a camera with different properties. None of the solutions I found online seemed to work on a mobile.

The ugly solution My logic was that a raycast originating from the cursor (center) should aim at the same direction no matter the camera projection matrix. So when a user clicks anywhere on the screen we can send a raycast from the cursor and then calculate the offset using the collision point, distance and angle.

  1. Create a large plane as a child of the <a-marker and give it the .collidable class
  2. When the user clicasdks on the screen send a raycast from the cursor
  3. Detect the collision point on the plane
  4. Calculate the actual desired collision position (get where the user clicked on the screen, multiply it by a distance factor and place it on the plane accordingly - this is your raycast hit)
  5. Cycle through your real collidable objects and check if the distance between the new hitpoint and the the collidbale is bellow your threshold.

HTML:

    <a-entity camera>
        <a-entity id="ray-rot-x">
            <a-entity raycaster="objects: .collidable; showLine: false; far: 30; useWorldCoordinates: true"
                id="ray-rot-y"></a-entity>
        </a-entity>
        <a-entity cursor="fuse: false" position="0 0 -1"
            geometry="primitive: ring; radiusInner: 0.02; radiusOuter: 0.03"
            material="color: white; shader: flat; opacity:.05">
        </a-entity>
    </a-entity>

JS:

AFRAME.registerComponent("raycaster-listen", { init: function () { let el = this.el let myRaycaster let canvas = this.el.sceneEl.canvas; let clickPosUpdated = new THREE.Vector3(); let distanceMultiplierX = .6; let distanceMultiplierY = .45; let isMobile = this.el.sceneEl.isMobile; this.el.addEventListener("raycaster-intersected", evt => { this.raycaster = evt.detail.el; myRaycaster = this.raycaster; }); this.el.addEventListener("raycaster-intersected-cleared", evt => { this.raycaster = null; myRaycaster = null });

  canvas.addEventListener("touchstart", function (evt) {
      console.log(evt)
      // GET MOUSE XY
      var _canvasSize = canvas.getBoundingClientRect(),
          w = _canvasSize.width,
          h = _canvasSize.height,
          offsetW = _canvasSize.left,
          offsetH = _canvasSize.top;
      var cx = void 0,
          cy = void 0;
      if (isMobile) {
          var touches = evt.touches;
          if (!touches || touches.length !== 1) {
              return;
          }
          var touch = touches[0];
          cx = touch.clientX;
          cy = touch.clientY;
      } else {
          cx = evt.clientX;
          cy = evt.clientY;
      }
      cx = cx - offsetW;
      cy = cy - offsetH;
      var mouseX = (cx / w) * 2 - 1;
      var mouseY = -(cy / h) * 2 + 1;

      // RAYCAST
      let intersection = myRaycaster.components.raycaster.getIntersection(el);
      if (!intersection) {
          return;
      }

      // CALCULATE UDPATED POS
      clickPosUpdated.x = intersection.point.x + intersection.distance * distanceMultiplierX * mouseX;
      clickPosUpdated.y = intersection.point.y + intersection.distance * distanceMultiplierY * mouseY;
      clickPosUpdated.z = intersection.point.z; // ToDo - calculate the Z offset based on the angle of the collision

      // SET THE DEBUG BOX TO SEE WHERE IT HIG
      debugBox.setAttribute("position", clickPosUpdated);

  });

} });

Left to do: calculate the Z offset based on the angle of the collision / rotation of the marker.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jeromeetienne/AR.js/issues/148#issuecomment-606470700, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJTLTNWYIPRSSWW76IBZTRKGQXJANCNFSM4DVKNSBA .

kvabakoma commented 4 years ago

What’s the difference about the ar.js camera? - I never found out. I based my assumption on jeromme etiene's comment saying that this is "due to the camera having a custom projection matrix.".

Has the cursor ever worked? - the cursor seems to work fine if it is not setup with rayOrigin: mouse

When did it break? - I've only tried it in the past month

I had a job for a client where users had to tap on moving objects in arjs. The default a-frame implementation (cursor="rayOrigin: mouse") did not work with arjs. I looked all over the Internet for solutions and did not find anything working on mobile (there was smth for desktop though). This topic was the closest I got to an explanation or solution to the issue.

In one of the earlier comments jeromme etiene says that this is "due to the camera having a custom projection matrix.". Based on this I tried to inspect the three scene and see if I can do anything about the camera. I failed and never really got to the bottom of the problem.

Pressed by the project's timeline I tried a few workarounds and the one I've written above gave decent results so I shared it :)

nicolocarpignoli commented 4 years ago

@kvabakoma Hi There! I tried your code.

There is debugBox variable not specified when trying to access to it (I think a querySelector is missed). I have to dig on it better, in general. Did you have finished it?

Also, I suggest you all to continue discussing this on this PR, on the new Repo, this one has been deprecated: https://github.com/AR-js-org/AR.js/issues/66

kvabakoma commented 4 years ago

@nicolocarpignoli thanks for pointing out the code ain't working. It seems I've put a wrong version. I've updated my previous post with what should be working. You can also see a demo here: link - it uses ontouchstart so you should test it on mobile or in dev tools by emulating a phone. Marker: https://github.com/artoolkit/artoolkit5/blob/master/doc/patterns/Matrix%20code%203x3%20(72dpi)/5.png Will share it on the other thread as well. Cheers :)