Face rotation with rear camera and both tracking

[AdrienMgm]

Hello @tdmowrer, @jimmy-jam,

I am trying to use WorldTracking and FaceTracking in the same time with rear camera feed from ARKit3 via ARFoundation.

The goal is to get eye pose (relative to face) from the FaceTracking to interact in the WorldTracking space. Like a laser controlled by our eyes (face pose + fixationPoint pose) to target some objects in the WorldTracking space.

But it seems that the face rotation is not correct. Whereas when I tried only with FaceTracking and front face camera feed, the rotation seemed to be completely normal and as expected based on our face movements.

From Apple documentation:

var userFaceTrackingEnabled: Bool A flag that determines whether ARKit tracks the user's face in a world tracking session.

and

The inherited transform property describes the face’s current position and orientation in world coordinates; that is, in a coordinate space relative to that specified by the worldAlignment property of the session configuration.

So I guess I have to make some transformations to get face and eyes pose correctly in the WorldTracking space. My project has a Content Placement Offset gameObject due to previous calls of MakeContentAppearAt() and a scale applied to the ARSessionOrigin. But despite all my attempts I didn’t succeed to get the right rotation and when I logged the raw values, they were really weird.

Could you please take a look at this specific case to see if it's a bug or if I have made some mistakes on my side.

---

Unity 2019.2.3f1 Xcode 11 beta 7 iOS 13 beta iPad Pro (3rd generation) 12.9-inch ARFoundation 3.0.0 preview 1 ARKit Face Tracking 3.0.0 preview 1 ARKit XR Plugin 3.0.0 preview 1

[karashinasou]

Hello. I suffer from the same phenomenon as AdrienMgm. Please confirm. My development environment is as follows.

---

Unity 2019.2.0f1 Xcode 11 beta 7 iOS 13.1 beta iPhone XS ARFoundation 3.0.0 preview 1 ARKit Face Tracking 3.0.0 preview 1 ARKit XR Plugin 3.0.0 preview 1

[AdrienMgm]

Any news @tdmowrer, @jimmy-jam ?

[tdmowrer]

Sorry for the delay. I am investigating this now.

The only difference between the front and rear camera modes is the way face tracking is enabled: as you noted, we use either an ARWorldTrackingConfiguration with userFaceTrackingEnabled or an ARFaceTrackingConfiguraton. Once a face is detected, it is processed in the same way. A quick debug log shows that the detected face has a non-identity rotation which updates in response to a moving face.

So I guess I have to make some transformations to get face and eyes pose correctly in the WorldTracking space.

No, you shouldn't have to do anything differently.

it seems that the face rotation is not correct

Can you elaborate? How are they incorrect?

[AdrienMgm]

I suppose that face pose should be handled differently if we are with the front facing camera or with the rear camera.

As Apple says in the doc, faces poses are relative to the world space and the world configuration:

The inherited transform property describes the face’s current position and orientation in world coordinates; that is, in a coordinate space relative to that specified by the worldAlignment property of the session configuration.

That's why I assume the transformation shouldn’t be the same in the two different context. However ARKit's Susbystem is precompiled and not opensource so we can't verify how the integration is done.

In the front tracking mode with front camera feed, the device is the "world center" of the coordinate space and faces are relative to this. In that case everything seems to be consistent.

In the both tracking mode with rear camera feed, the device is also the "world center" of the coordinate space until we offset it with the Content Placement Offset system and the MakeContentAppearAt() function.

By the way, I don't know if the tracking mode changes the rotation of the coordinate space. If this is the case, we should take care of that too.

So in my tests, the raw values returned from the ARFace GameObject (and the forward of the fixationPoint) aren’t the identity rotation but the values seem completely messed up. For example, rotating my head around the y-axis makes my debug axis GameObject rotate around another axis. Plus, when I rotate around the center of my scene, all the rotate axis meld together.

Thanks for your time.

[tdmowrer]

Face tracking in rear camera feed mode (i.e., ARWorldTrackingConfiguration with userFaceTrackingEnabled) has behavior for which we did not account.

In ARKit, every ARAnchor has a 4x4 transform matrix which describes its position and rotation. In an ARFaceTrackingConfiguration, an ARFaceAnchor is oriented as in the following diagram provided by Apple:

Note:

1. The red +X axis points to the right from the perspective of someone facing the head, but in the local space of the head, it points out of the face's left check. Since this head is a mirror image of the person holding the phone, this probably makes the most sense (i.e., my right cheek is on the right side of the image, but that corresponds to the head model's left cheek).
2. Like all Apple SDKs, this uses a right-handed coordinate system (+X crossed with +Y is +Z using the right-hand rule).

In ARFoundation, we use the Apple-provided function simd_quaternion(simd_float4x4 matrix) to extract the rotational part of the transform matrix to a quaternion, which we then provide to Unity. simd_quaternion assumes its input is a right-handed rotation matrix.

When using an ARWorldTrackingConfiguration with userFaceTrackingEnabled, the transform is uniquely different. The transform matrix associated with the ARFaceAnchor is left-handed, not right-handed. That is, +X cross +Y is -Z using the right-hand rule. I suspect they are attempting to negate the X axis in the above diagram to point the other way, without affecting the other axes. In rear-camera mode, your face is not a mirror image but should be accurately represented, so +X should point to the right in face space and the other axes should not change. However, this gives you a left-handed matrix, and passing this matrix to simd_quaternion now fails to produce the correct rotation because the input is no longer right-handed. As far as I can tell, this distinction is not documented.

This is the source of the problem you are seeing: we are providing a left-handed rotation matrix to simd_quaternion which expects a right-handed matrix.

This will be fixed in the next release. You can work around it now with a bit of Objective-C code in your project:

#import <ARKit/ARKit.h>

typedef struct
{
    int version;
    void* anchor;
}  NativeFace;

simd_float4 GetRotation(NativeFace* nativeFace)
{
    ARFaceAnchor* faceAnchor = (__bridge ARFaceAnchor*)nativeFace->anchor;

    // Flip handedness
    const simd_float3x3 rotation = simd_matrix(
         faceAnchor.transform.columns[0].xyz,
         faceAnchor.transform.columns[1].xyz,
        -faceAnchor.transform.columns[2].xyz);

    // Convert to quaternion
    const simd_float4 v = simd_quaternion(rotation).vector;

    // Convert back to left-handed for Unity
    return simd_make_float4(v.xy, -v.zw);
}

Then in C#:

using System;
using System.Runtime.InteropServices;
using UnityEngine;

...

[DllImport("__Internal")]
extern Quaternion GetFaceRotation(IntPtr ptr);

void HandleFaceRotation(ARFace face)
{
    var rotation = GetFaceRotation(face.nativePtr);

    // do something with rotation
}

[AdrienMgm]

Thanks for your feedback, indeed the left-handed transform of the ARFaceAnchor was the main bug.

However, when I tried your workaround it didn’t work. After some digging, I finally found that the bridge between the Objective-C plugin and the C# bridge didn’t cast/map the values correctly.

In your sample simd_float4 GetRotation(NativeFace* nativeFace) return an simd_float4 (that represent 4 floats) in memory and the extern Quaternion GetFaceRotation(IntPtr ptr) return a Unity Quaternion. From Objective-C plugin the values were consistent but once logged in Unity the Quaternion's values were wrong. I didn’t really understand why it fails to cross over to Unity. Maybe the casting/mapping done by Unity from the plugin part doesn’t work because of the structs differences between simd_float4 and Quaternion.

Anyway, I got everything working correctly by adding another struct in the plugin and now Unity seems to be able to map everything correctly.

Here the code for those interested:

Objective-C:

#import <ARKit/ARKit.h>

typedef struct
{
    int version;
    void* anchor;
}  NativeFace;

typedef struct
{
    float x,y,z,w;
} UnityARVector4;

extern "C"
{
    UnityARVector4 GetFaceRotation(NativeFace* nativeFace)
    {
        ARFaceAnchor* faceAnchor = (__bridge ARFaceAnchor*)nativeFace->anchor;

        const simd_float3x3 rhtrans = simd_matrix(
            faceAnchor.transform.columns[0].xyz,
            faceAnchor.transform.columns[1].xyz,
            -faceAnchor.transform.columns[2].xyz);

        // Convert to quaternion
        const simd_float4 v = simd_quaternion(rhtrans).vector;

        // Convert back to left-handed for Unity
        UnityARVector4 lhv = {v.x, v.y, -v.z, -v.w};
        return lhv;
    }
}

C#:

using System;
using System.Runtime.InteropServices;
using UnityEngine;

public class ARKitCustomBridge
{
    #if UNITY_IOS
        [DllImport("__Internal")]
        public static extern Quaternion GetFaceRotation(IntPtr facePtr);
    #endif
}

[tdmowrer]

Fixed in 3.0.0-preview.4

[teobabic]

@tdmowrer Is this fix only applied to the ARFaceAnchor or all ARFaceManager.trackables? In a face tracking in rear camera feed mode (i.e., ARWorldTrackingConfiguration with userFaceTrackingEnabled), I found the same issue with the rotation of the ARFace.leftEye or rightEye.

Apple ARKit leftEyeTransform: https://developer.apple.com/documentation/arkit/arfaceanchor/2968191-lefteyetransform?language=objc

Unity AR Foundation 3.0.1: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@3.0/api/UnityEngine.XR.ARFoundation.ARFace.html#UnityEngine_XR_ARFoundation_ARFace_leftEye

[tdmowrer]

The fix is not applied to the left/right eye because it is not necessary. When performing face tracking with the rear-facing camera active, ARKit uses a left-handed transform for the ARFaceAnchor, but the eye transforms continue to use a right-handed transform, so the issue discussed in this thread does not apply to the eye transforms.

I also modified the sample in this repo to copy the eye transforms to a cube and they look correct. Can you describe the issue you are seeing?

[teobabic]

Thanks for your feedback, indeed the right-handed transform of the eyes was the problem.

After some testing I was able to re-create the wanted eye transforms. When we use simultaneous tracking world- and face-tracking, are the eye transforms provided in relation to the session space and not the face pose itself (as when only using face-tracking). With few Transform.InverseTransformPoint (for position), Quaternion.Inverse (for rotation) commands, and by flipping some axes (for example: -x, -y, +z) to change the right- to left-handedness, we can create eye transforms which are in relation to the face pose itself (childs of the face anchor).

[andrewcccc]

Hi @tdmowrer! I believe that is happening with Swift as well. Is there a version to fix the coordinate system in Swift?

[tdmowrer]

Hi @tdmowrer! I believe that is happening with Swift as well. Is there a version to fix the coordinate system in Swift?

I'm not sure I understand the question. What does Swift have to do with this issue?

[andrewcccc]

Hi @tdmowrer. Thank you so much for the response. I am using ARkit in Swift and the TrueDepth camera for face tracking: https://developer.apple.com/documentation/arkit/tracking_and_visualizing_faces (as in this example). However, I noticed that I am getting negative z axis which I believe that its ARFaceAnchor is left-handed (not as described in the document). I am wondering how is it possible to do the similar correction to right-handed in Swift as you did it in unity. Thanks for your time!

[tdmowrer]

It sounds like you are writing a native app using Swift. This issues page is for questions about Unity's ARFoundation and its associated support packages. It's not a place to ask general native ARKit questions. Have you tried Apple's ARKit forum?

[elnazta]

@tdmowrer I know you closed this, but I am using ARKIT XR Plugin 4.1.0, I also have the same version ARKIT Face Tracking installed in Unity. I am using AR Foundation 4.1.0 and unfortunately I can't find out how you will have both the user facing camera and world facing camera simultaneously working, while looking at the feed from the world camera. For instance, what is the correct way of getting the Face position, rotation and eye positions of the user facing camera while having the world camera active? Most references and tutorials I can find that will help me enable both the world tracking configuration (I'm assuming using Plane detection added to the AR Session origin in the current version) and the Face tracking configuration (I'm assuming by adding the AR Face manager to AR Session origin) are all based on the ARKIT deprecated plugin. I would appreciate if you can respond to this.

[tdmowrer]

@tdmowrer I know you closed this, but I am using ARKIT XR Plugin 4.1.0, I also have the same version ARKIT Face Tracking installed in Unity. I am using AR Foundation 4.1.0 and unfortunately I can't find out how you will have both the user facing camera and world facing camera simultaneously working, while looking at the feed from the world camera. For instance, what is the correct way of getting the Face position, rotation and eye positions of the user facing camera while having the world camera active? Most references and tutorials I can find that will help me enable both the world tracking configuration (I'm assuming using Plane detection added to the AR Session origin in the current version) and the Face tracking configuration (I'm assuming by adding the AR Face manager to AR Session origin) are all based on the ARKIT deprecated plugin. I would appreciate if you can respond to this.

This thread is already pretty diverse, and you're asking about a newer feature in 4.x. Would you mind starting a new thread?