search

jeromeetienne / AR.js

Efficient Augmented Reality for the Web - 60fps on mobile!
MIT License
15.8k stars 2.22k forks source link

Marker based and Location based elements does not work together #702

Closed umutto closed 4 years ago

umutto commented 4 years ago

Describe the bug When using geo based and marker based AR in the same scene only either one of them work. Problem seems to be the camera entity or I am missing something.

When <a-entity camera/> is used as advised on arjs docs for multiple marker support, geo based elements are not displayed. <a-entity camera gps-camera rotation-reader/> does not work either.

When <a-camera gps-camera rotation-reader/> is used as advised on geoarjs docs, then barcode or pattern marker is found but the overlaying a-element can't be seen (I was able to see it at some weird lucky angles, maybe the problem is due to camera being moved?). Acts as same when <a-camera camera gps-camera rotation-reader/> is used or when 2 different camera elements used as <a-camera camera gps-camera rotation-reader/><a-entity camera/>

To Reproduce I have prepared a simple example to reproduce, change the camera elements to see different cases:

<!DOCTYPE html>
<script src="https://cdn.jsdelivr.net/gh/aframevr/aframe@1c2407b26c61958baa93967b5412487cd94b290b/dist/aframe-master.min.js"></script>
<script src="aframe-ar.min.js"></script>
<script>
    window.onload = () => {
        let scene = document.querySelector('a-scene');
        if (navigator.geolocation) {
            navigator.geolocation.getCurrentPosition(function(position) {
                let box = document.createElement('a-box');
                box.setAttribute('gps-entity-place', `latitude: ${position.coords.latitude - 0.001}; longitude: ${position.coords.longitude +0.001}`);
                box.setAttribute('scale', '20 20 20');
                box.setAttribute('color', 'red')
                scene.appendChild(box);
            });
        }
    }
    AFRAME.registerComponent("markerdebug", {
        init: function() {
            this.el.addEventListener("markerFound", function(e) {
                console.log('found')
            });

            this.el.addEventListener("markerLost", function(e) {
                console.log('lost')
            });
        }
    });
</script>

<body style='margin : 0px; overflow: hidden; font-family: Monospace;'>
    <a-scene embedded markerdebug arjs='sourceType: webcam; detectionMode: mono_and_matrix; matrixCodeType: 3x3; debugUIEnabled: false;'>

        <a-marker preset='hiro'>
            <a-sphere position='0 0.5 0' material='opacity: 0.5; side: double;color:blue;'>
            </a-sphere>
        </a-marker>

        <!-- Location based elements are displayed, hiro marker is found but element is not displayed -->
        <!-- <a-camera  gps-camera rotation-reader/> -->

        <!-- Location based elements are displayed, hiro marker is found but element is not displayed -->
        <!-- <a-camera camera gps-camera rotation-reader/>
        <a-entity camera/> -->

        <!-- No gps based elements displayed, hiro marker is found and it's element is displayed -->
        <a-entity camera gps-camera rotation-reader/>
    </a-scene>
</body>

</html>

Expected behavior As my understanding of ARjs 2.0, both cases should be able to work at tandem. When a marker is read, it should display an element and when the device orientation changes to a nearby geo-entity-place it should display it's element. (sphere and a box on the example code)

Desktop (please complete the following information):

Smartphone (please complete the following information):

Thank you very much!

nicolocarpignoli commented 4 years ago

Hi, thanks, that's actually a bug.

I gave a look, and two problems turned out:

1) a-camera and a-entity camera cannot be used together (so have two of them), and we have to find a way to make a-camera behave like a-entity camera (or vice versa)

2) By removing this piece of code, location-based feature breaks. To not have it, in some cases (e.g. local machine) breaks marker-based:

 <script>
        THREEx.ArToolkitContext.baseURL = 'https://raw.githack.com/jeromeetienne/ar.js/master/three.js/'
    </script>
nicolocarpignoli commented 4 years ago

@umutto feel free to give it a look and to open a PR on this :)

umutto commented 4 years ago

I would like to but after looking into it I'm a bit lost. I'm not very familiar with three.js and 3D calculations so I don't think I would be a great help.

Saying that I think the reason is in marker based recognition, camera is set to static (in <0, 0, 0>), never moves and marker object positions are updated accordingly. This is also written in the documentation here (a-marker-camera).

Now, there is a changeMatrixMode parameter that is set to modelViewMatrix as default (where camera is static). But changing it to cameraTransformMatrix (camera is movable but markers are static) breaks it.

So I've looked into offsetting the markers to kinda move together when gps-camera moves by adding the following lines here.

if (document.querySelector('[gps-camera]')) {
    var cameraPosition = document.querySelector('[gps-camera]').object3D.position;
    markerObject3D.position.add(cameraPosition)
}

which fixes the positioning, but due to my inexperience in 3D calculations I couldn't offset the camera rotation that happen here. Also this kinda felt a bit hackish way.

I think we should adapt geoAR.js to static camera or fix the cameraTransformMatrix code and adapt AR.js marker tracking to movable camera (latter sounds more clean imho). But doing which is above my experience.

umutto commented 4 years ago

Actually now that I had a second look, setting changeMatrixMode to cameraTransformMatrix does not break marker recognition (only some parts does not work but main function works, broken parts are markers object3d stays visible, it flickers, does not emit events etc..). As expected marker stays static and camera moves. But camera's position doesn't change as I assumed but rather camera.getWorldDirection() changes (now that I think of it is obvious.)

I'm not sure how to combine compass heading and this camera rotation but solution seems to be that. Sadly I won't have much time to work on this couple of weeks, but I'll try to take a look after that.

nicolocarpignoli commented 4 years ago

Thank you. I will be busy for nft development of arjs, so our best bet for now is you.😊

andypotato commented 4 years ago

Saying that I think the reason is in marker based recognition, camera is set to static (in <0, 0, 0>), never moves and marker object positions are updated accordingly. This is also written in the documentation here (a-marker-camera).

I'd also love to see this feature implemented, so I had a quick look at the source code. In my opinion with the current implementation this is unfortunately impossible. As @umutto points out, the camera behavior in both "modes" is fundamentally different.

The approach I would suggest is to render both Three.js scenes and then overlay them on the same canvas. Means there are two active cameras, one handling markers, one handling location based objects. Both renderers then render into the same canvas resulting in an overlayed picture.

@nicolocarpignoli @umutto Thoughts? Comments?

nicolocarpignoli commented 4 years ago

@andypotato it's a brilliant idea. Maybe the only that can work without have to rewrite one of the two features.

umutto commented 4 years ago

@andypotato Yeah good observation and idea! The two features are fairly separated, I don't think they use any common code so it should be most straight forward / quick way to implement it. I doubt it but wonder if that would effect the performance though.

Sadly I will not have time to check it out until next month, but would gladly help/test after then.

nicolocarpignoli commented 4 years ago

added on new repo, we can continue there: https://github.com/AR-js-org/AR.js/issues/29