Frame Rate Drops to 20 when using Face Tracking

[shubhamguptak]

Hi I build an App with both Ground Tracking and Face Tracking and made a button which switches between them. The Problem is with the performance the Ground tracking which works on Rear camera gives 30 fps where as when i am using Face Tracking which uses Front Camera the fps drops to 20 fps which gives a lot of lag. Can you please help how should i increase the fps of the front camera face tracking. Unity 2019.2.11f1 ARFoundation Version - 3.0.1

[tdmowrer]

What device? My experience on Android is face tracking can take upwards of 45 ms per frame (about 20 fps) on older devices (e.g., a first generation Pixel). There's nothing that can be done about that from either your or our (Unity's) end; it's a very CPU intensive feature within ARCore.

If that doesn't sound right, please provide details on your device and a profiler capture of a few frames.

[shubhamguptak]

I am testing on Xiaomi POCO F1. It has 6 GB of ram and Snapdragon 845 processor. The problem is solved i had some process running on the background that was dropping the fps. Now I am getting about 33ms (30 fps) on the device. Also, I want to know how can we get the feature points such as nose, eyes, ears, mouth and can use those to instantiate a prefab only there like earrings for ears, sunglasses for eyes, etc.

[tdmowrer]

ARCore gives you "face regions" which is currently nose, left forehead, and right forehead. This repo contains a sample that demonstrates their use. See https://github.com/Unity-Technologies/arfoundation-samples/blob/master/README.md#arcorefaceregions

[shubhamguptak]

It's giving me 20fps when I checked again. Definitely Something weird is happening

[tdmowrer]

Can you provide a profiler capture?

[shubhamguptak]

Here's the profile capture https://imgur.com/a/lT6A2NY and also the data file https://drive.google.com/file/d/1HdEUlqrouYKVVW4SfWOqTfwxkixat8H6/view?usp=sharing

[shubhamguptak]

[tdmowrer]

Thanks, that is helpful. It seems like there's a lot going on in your app besides just AR, so I think you should focus on some of the spikiest frames and try to see where the time is going. Remember that you can add custom profiler markers to your own code. Some things that stand out to me:

Time spent in AR The bulk of the ARSession.Update time is spent stepping ARCore, and looks to be about ~20ms per frame. Again, face tracking is one for the more CPU intensive features on ARCore, so that time seems plausible to me.

Rendering

• Rendering takes ~10ms with spikes up to 25ms

  

• FpsDisplay.OnGUI seems to generate a lot of garbage (GC allocations) and some spikes are caused by the garbage collector.

  

• ARFace.Update normally doesn't take much time but there are a few frames where it spikes to 10+ms. That method just invokes its event listener -- have you subscribed to this event? Note that the ARFaceMeshVisualizer also subscribes to this event in order to update the render mesh, but that should be consistent.

[shubhamguptak]

Ok i can remove the FPS display. So in my app i have created one toggle button that switches from ground plane to face detection. I have two arsession script one is for Face tracking and other for Ground tracking as you can see in the below image. So with the toggle button I am just enabling and disabling the Parent prefab (Ground plane) and (Face tracking). Is it a good practice or i should instantiate and destroy the parent prefabs based on my needs. I think keeping both

[tdmowrer]

I think your setup is fine. This doesn’t explain the frame spikes, though. Did my comments help you track down where the time is going?

[shubhamguptak]

yes it did helped. Thanks a lot