Augmented Faces: regions (nose, forehead left/right) aren't steady steady enough, perceived pose estimation latency

[AndroidDeveloperLB]

Tested on Pixel 2 with Android P. PQ2A.190205.002 There is about tiny delay between what I do and what I see on the sample. It's small, but still very noticeable. To reproduce it easily, try to move your head left and right quickly.

Also, the masking (ears and nose in this sample) seem to move a lot even though I stand still.

[shliama]

Yeah, same here with Pixel 2 XL. Even when the head & device is steady, and the light is "perfect" - ears and nose jitter noticeably.

[michaelvogt]

Well, this is what ARCore is able to deliver. Reported many times, no improvement since the beginning.

[AndroidDeveloperLB]

@michaelvogt How come? This is a good device. If it works fine on many apps, it should work fine here too. Are there any plans to improve it?

[michaelvogt]

The experience so far was, that the ARCore team stays silent about these questions. The same way it was with Tango. They are definitely aware of the problems, but chose so far to do nothing.

No, I'm not part of the team, so I don't know more than you. It's really time that they get their act together and start to improve communication and functionality.

[fredsa]

@AndroidDeveloperLB can you share a screen recording or what you're seeing?

1. Use adb shell screenrecord --bugreport /sdcard/capture.mp4 to begin capturing the video
2. Hit CTRL-C once your done capturing video.
3. Wait briefly for the recording to finish
4. Use adb pull /sdcard/capture.mp4 to get the file off of phone

[AndroidDeveloperLB]

@fredsa No, for 2 reasons:

1. It doesn't make sense, because all you will see it myself with the mask of the sample, while what you are supposed to see is me AND what's on the device in sync, seeing the difference in time between them. So what I will need is 2 smartphones. One of them somehow recording both me and the smartphone with the app at the same time (which is quite impossible on its own because of the needed angle). Plus, it will be extremely hard to do it with just 2 hands.
2. I don't want to show myself here.

[michaelvogt]

@fredsa

Hello, appreciate your response. Since this problem exists since Tango, the are several screen recordings here on github.

can you share a screen recording or what you're seeing?

Does this suggest, that the team still wasn't able to reproduce the instability problem of ARCore? I can't imagine this, because the problem is so bad, there is no way you never saw this yourself.

[michaelvogt]

@fredsa

Have fun: https://photos.app.goo.gl/BhudFXPZuxtGnRNu9

Ignore the layering problem of the right image, that might be solved by now.

Other problems you'll see is, shadow and plane is drawn on top of the background. Not commented, nor fixed till today. That definitely is a Sceneform problem.

Oh, there was one! comment, that the background can't be at 0,0,0 on top of the image. That is definitely not the problem.

As you can see, the weather situation is close to optimal. No problem from this side.

You'll find an issue discussed with @inio at ARCore before. That problem doesn't apply here.

I'm absolutely sure, looking at this, you have to admit that this is absolutely unacceptable, completely useless. Both the problem and that it's ignored by the team for such a long time.

I believe I had enough patience with this. It's time that's someone is finally looking into it, and not keep ignoring it, giving the impression every thing is ok. It is just perfect that the same problem still exists with the face tracking. A problem already reported with Tango.

Yes I know Tango isn't Google code. But Google should at least be willing to fix the problems it had, still has even with a different product name.

[fredsa]

@AndroidDeveloperLB understood.

@michaelvogt Michael, thank you for sharing the video. Its appears to be one of "Augmented Images" rather than "Augmented Faces". Nonetheless, I'll share it with the team. It would be helpful to know what device you were running on and what version of the O/S and ARCore were used to build and run the app.

The reason videos and specifics about the device and software versions are especially useful is that it helps us diagnose and understand specific tracking issues you're seeing, with the aim of improving real world tracking performance in future updates. Videos really help us to identify the cause of the tracking issues reported here, since there are several potential causes:

• Hardware/device specific issues
• Software specific issues; includes drivers, O/S versions, and obviously the installed version of ARCore
• Environmental issues; lighting; degree of motion; amount of visible detail
• App specific issues (e.g. placing objects in the world without anchors, which can lead to visible apparent drift compared to using anchors)
• etc.

[michaelvogt]

Steadiness of tracking shows the same negative behavior, no need to be picky.

Yes, problems to such extend are not always shown. At that time, I could observe this erratic behavior for over 2 hours (then the batteries of both phones where empty).

Yes, I heard lot's of creative possible causes for the problem. Finally starting to improve the situation would be good.

Just right now, another issue of unsteady tracking came in.

The phone used was a Pixel and Zenfone AR. Both showed exactly the same problems. Software was the latest at recording time. But, as I said, I see such behavior with loads of different apps since Tango. The problem is inherent in the algorithms you use.

What I unfortunately don't have on video is, how ARCore starts to create 3, 4 ground planes on top of each other, then suddenly the dots came flying towards the camera. And this was with Hello Sceneform app.

I can't imagine, that you aren't able to reproduce any tracking errors.

[fredsa]

@michaelvogt Appreciate the feedback. I hear you when you say you'd like to see an overall improvement in tracking performance across the different features provided by ARCore.

I'd like to keep this issue focused on the above reported concerns: steadiness and perceivable pose estimation latency of the provided face regions (nose, forehead left, forehead right).

[michaelvogt]

Understood, its an ARCore problem anyway. But still its clear, that the augmented images aren't usable as it is right now.

I even think that the team should be working on better localisation features anyway - making the cloud anchors actually usable, providing something that could be called an AR cloud.

[AndroidDeveloperLB]

Is this issue occurring on the other types of libraries (Unity and others) ?

[michaelvogt]

In case the problem is in the ARCore algorithms (which I assume), I'd expect that the same problems show up anywhere. As I don't use unity or the iOS libraries, I don't know.

[AndroidDeveloperLB]

Just wanted to make sure it's ARCore issue...