Aframe raycaster broken when AR.js is imported (click detection not working correctly)

vvigilante commented 3 years ago

Do you want to request a feature or report a bug? bug

What is the current behavior? The "clickable" box does not correspond to the place where the object is displayed on screen, i.e. sometimes there is a good overlap (but still, some part outside the object still counts as clickable, while some part of the object does not react). With mouse and keyboard this is evident by looking at the cursor change

If I fire up the inspector (ctrl+alt+i) clicking on the box inside the inspector actually works perfectly. If I do not load AR.js but I just load aframe, the raycaster works perfectly too. Tested with aframe 0.9.2 and 1.2.0

If the current behavior is a bug, please provide the steps to reproduce.

I used the code from here, just with a smaller box. Here is the code:

<!DOCTYPE html>
<html>
<head>
    <meta charset='utf-8'>
    <meta http-equiv='X-UA-Compatible' content='IE=edge'>
    <title>GeoAR.js demo</title>
    <script src="https://aframe.io/releases/0.9.2/aframe.min.js"></script>
    <script src="https://raw.githack.com/AR-js-org/AR.js/3.3.3/aframe/build/aframe-ar.js"></script>
</head>

<script>
    AFRAME.registerComponent('clickhandler', {
        init: function() {
            this.el.addEventListener('click', () => {
                alert('Clicked!')
            });
    }});
</script>

<body style='margin: 0; overflow: hidden;'>
    <a-scene
        cursor='rayOrigin: mouse; fuse: false'
        raycaster="objects: [clickhandler];"
        vr-mode-ui="enabled: false"
        embedded
        arjs='sourceType: webcam; debugUIEnabled: false;'>

        <a-box
            clickhandler
            material="color: red;"
            scale="1 1 1"
            position="0 1 -15">
        </a-box>

        <a-camera gps-camera rotation-reader>
        </a-camera>
    </a-scene>
</body>

Please mention other relevant information such as the browser version, Operating System and Device Name Chrome on Windows 10 and Android 10

What is the expected behavior? The object should trigger the click event when I click inside the displayed object.

More info / speculation: The problem seems to be related to the fact that the camera image does not match the aspect ratio of the browser viewport, so the canvas gets resized accordingly. On my desktop for instance the problem concerns mainly the vertical axis (i.e. the horizontal detection is correct) and this may be related to the fact that the camera image is taller than the screen is.

vvigilante commented 3 years ago

Probably this is the same problem as here: https://github.com/AR-js-org/AR.js/issues/40. Sorry for opening a duplicate issue

nickw1 commented 3 years ago

Hi @vvigilante I think this might be due to changes in A-Frame.

Look at the source for this page. https://hikar.org/webapp/ex1.html

Note how the raycaster is setup in a different way and near and far properties are set on the raycaster (it does not work without these).

Does using a variation of that example work for you?

vvigilante commented 3 years ago

Hi @nickw1, thank you for your help. I confirm that the example you suggest is working. I believe this is not due to a different version of aframe nor to the near/far setting, but rather to the setting videoTexture: true of ar.js, which prevents the 3d canvas from being resized and stretches the image to fit the window.

If you just remove the setting videoTexture: true, the problem I described comes right back. I believe that the solution with the stretched image is not acceptable.

If we remove both far and near from the raycaster, the click detection stops working altogether, but I believe that is a different problem.

nickw1 commented 3 years ago

@vvigilante not using videoTexture: true has its own problems, indeed I contributed that around a year ago because the original solution was preventing POIs more than around 1-2km from showing even if the camera far parameter was set to further than this. There is likely to be a fix to videoTexture which prevents the stretching as you describe, for example selecting a camera dimension that most closely fits the aspect ratio of the window.

The near/far issue has come up before and may be due to changes in A-Frame but I am not sure. The documentation suggests these are not necessary, I have to admit I'm not sure of the cause of this.

vvigilante commented 3 years ago

Ok: news. Setting the camera to fov:40 appears to work. The default fov is 80.

This is what I observe

The raytracer always works according to the camera parameters
If AR.js is included and videoTexture is set to true, the displayed camera has correct position but fov=40. Opening the inspector (ctrl+alt+i) and changing the camera fov has no effect whatsoever in this case.

So the raytracer always works consistently, but the field of view displayed in the canvas is the culprit.

Do you know if this override of the fov is done intentionally inside AR.js? If so, why is that?

nickw1 commented 3 years ago

@vvigilante Interesting - I was unaware of the fov being set to 40 within AR.js, or whether it is even done within AR.js; maybe @nicolocarpignoli might have an idea.

I had noticed that the camera feed and augmented content did not seem to 'fit' very well in the distance, as if the camera feed was being stretched somehow as you observe. My device has a fov very close to the default of 80 and I was a little puzzled by this, will be interesting to see if setting the fov to 40 corrects it. Maybe it's just using the wrong preview dimension?

AR-js-org / AR.js

Aframe raycaster broken when AR.js is imported (click detection not working correctly) #271