[Question] How to find out if an image anchor has been removed?

[oOMoeOo]

Environment

Please provide the following information about your environment:

1. Development OS: Mac
2. Device OS & Version: iOS 11.4, 12.0 beta
3. Version: ViroReact 2.10.0 and React Native 0.55.4
4. Device(s): iPhone X, iPhone 7

Description

1. Is there a chance to get notified when an image anchor has been removed or is not in the view anymore? I tried the onAnchorRemoved function but had no luck. The function is never called.
2. onAnchorUpdated is called for image anchors even if the anchor hasn't moved what leads to a jittering effect. Is that a problem with viro-react or ARKit 1.5 itself?
3. When using image anchors only the corresponding methods like onAnchorFound/Updated/Removed pass the anchor. Is this cases the anchorType (anchor.type) is "anchor" instead of "image".
4. When using multiple image markers: Is it possible to find out which one was detected when onAnchorFound has been called?

Thanks Moe

[achuvm]

Hi,

1. onAnchorRemoved() should be called when the anchor is removed by the underlying system, but it won't get called when a image is simply "out of view". We don't currently have a good way to detect that.
2. onAnchorUpdated() is invoked/called by the underlying AR system (ARKit). Over time, it does make minute changes to the position/rotation either because the image itself moves or ARKit has a better estimation of the user's own position/rotation and so the position/rotation of the anchor needs to be updated.
3. That's an interesting find. Are you using the onAnchorFound|Updated|Removed on the ViroARScene or on the ViroARImageMarker components themselves? We'll have to investigate this one more.
4. In this case, I'd attach an onAnchorFound to the ViroARImageMarker component and from that, you know which one has been found.

Thanks for trying out Viro!

[oOMoeOo]

Hi,

do you have any ideas on how to detect whether an image marker is out of view? I'm trying to improve the ar precision by using a mixed marker vs. plane model where objects will bind to a plane in case the marker is currently out of view.

Thanks Moe

[achuvm]

Hi @oOMoeOo,

There's not a great way to do this as the AR subsystem doesn't let us know when the marker is no longer "visible". However, what you could do is compute the angle between the camera forward and the last known position of the image marker relative to the camera. If that's greater than say 45 degrees, then most likely the image marker is no longer in the viewport of the camera (as the camera FOV is likely less than that). The FOV of the camera would be per-device dependent if you'd like to be more accurate in your computation (vertical FOV can be different than the horizontal FOV too).

[oOMoeOo]

@achuvm Well, i'm wondering why neither arkit nor arcore provide this function. I think it shouldn't be so complicated to determine it in the underlying arkit/scenekit implementation :-)

We're developing an app that requires the user to draw shapes precisely using ar. We need the shape to be in place with a precision of 0-2mm even if the user moves around. This is not possible with plane detection only. That's why we provide the user with multiple image anchors that have to be placed on a surface with fixed distances. We then try to bind the shape to the anchor that is currently in the view (using the difference vector between two related anchors). This is where we need to determine the anchor to view relation. Isn't it just the same as testing if a geometry is in the view?

The idea of tracking the camera angle sounds good but it doesn't take the distance and position of the camera into account. Is there any chance to determine the FOV?

[achuvm]

Hi @oOMoeOo,

Yeah, that's a good point. You can try asking for a feature request :).

Yep, that's similar to testing if a geometry is in view. You can get the distance/position of the camera through the getCameraOrientationAsync() function on the ViroARScene component. We don't expose this information to the end-user right now. We might expose it in a future update or have a better way to determine if a geometry is in the viewport.

Thanks,

[pgims]

It seems it is now possible to determine in ARCore whether an image marker is currently detected in view or just tracked based on the last known location in the environment by using getTrackingMethod(). Is this already available in ViroReact? I can't find anything in the documentation. Is this planned for upcoming releases? This would be very helpful in our project.

Here are two links where this is mentioned:

https://developers.google.com/ar/develop/java/augmented-images#capabilities

Beginning with ARCore 1.9, you can use getTrackingMethod() to determine whether ARCore is currently tracking the image based on where the image is currently being detected in the camera view (FULL_TRACKING), or based on its most recently detected position in the world (LAST_KNOWN_POSE).

https://developers.google.com/ar/develop/java/augmented-images/guide#supporting_different_use_cases

When ARCore detects an Augmented Image, it creates a Trackable for that Augmented Image and sets TrackingState to TRACKING and TrackingMethod to FULL_TRACKING. When the tracked image moves out of camera view, ARCore changes the TrackingMethod to LAST_KNOWN_POSE while continuing to provide the orientation and position of the image.

[dthian]

Hey @pgims, as @achuvm mentioned in #682, we'll be exposing tracking methods in the next release.