Matching head poses with 3D landmarks

[rakadambi]

I read this issue here:

https://github.com/1adrianb/face-alignment/issues/165

I am trying to match head poses using 3D landmarks.

I have been successful in the 2D case.

I have 2 faces and I am trying to match the pose of one (in which the face can be looking in any direction) to another face which is frontal looking. I use Similarity Transformation matrix algorithm (_umeyama function from here: https://github.com/scikit-image/scikit-image/blob/master/skimage/transform/_geometric.py) with 2D landmarks to achieve this. This is working fine.

Now, I am trying to do the same with 3D landmarks. But for some reason it is not working. The same Umeyama algorithm gives me completely wrong results for rotation and translation. I am kind of stumped at this.

It doesn't seem like a difficult problem since I already have 3D landmarks points, thanks to Adrian. Has anybody done something like this before?

[rakadambi]

OK. I will try to solve this and document here. Let's see how it goes.

I have the src face which, say is a profile face and target face (call it tar face) which is a frontal looking face. The src face has dimensions of 1920x1080 and tar is 1024 and 1024.

I get 68 3D landmarks for both.

Now I pass noth src and tar landmarks here (_umeyama function from here: https://github.com/scikit-image/scikit-image/blob/master/skimage/transform/_geometric.py) and get a 4x4 homogeneous matrix. I then use "decomposeProjectionMatrix " and get the rotation and translation matrices.

Am I on the right track?

[rakadambi]

Some more experiments. I am now wondering if the 3D landmarks are correct? Let me try to illustrate with examples here: Attaching two images here. This is the source from which I am trying to match: This is the target:

I get 3D landmarks from both and just compute the roatation, specifically z axis rotation. This is the code to get the rotation and translation matrices. As you can see, the two images are fairly aligned already and the z axis angle should be very small.

The src argument is the 3D landmarks from eye corners [indices 36 and 45], nose tip [index 33], mouth corners [indices 48 and 54], and chin [index 8]. Same for tar landmarks.i

`

def umeyama( src, dst, estimate_scale ):
    num = src.shape[0]
    dim = src.shape[1]

    # Compute mean of src and dst.
    src_mean = src.mean(axis=0)
    dst_mean = dst.mean(axis=0)

    # Subtract mean from src and dst.
    src_demean = src - src_mean
    dst_demean = dst - dst_mean

    # Eq. (38).
    A = np.dot(dst_demean.T, src_demean) / num

    # Eq. (39).
    d = np.ones((dim,), dtype=np.double)
    if np.linalg.det(A) < 0:
        d[dim - 1] = -1

    T = np.eye(dim + 1, dtype=np.double)

    U, S, V = np.linalg.svd(A)

    # Eq. (40) and (43).
    rank = np.linalg.matrix_rank(A)
    if rank == 0:
        return np.nan * T
    elif rank == dim - 1:
        if np.linalg.det(U) * np.linalg.det(V) > 0:
            T[:dim, :dim] = np.dot(U, V)
        else:
            s = d[dim - 1]
            d[dim - 1] = -1
            T[:dim, :dim] = np.dot(U, np.dot(np.diag(d), V))
            d[dim - 1] = s
    else:
        T[:dim, :dim] = np.dot(U, np.dot(np.diag(d), V.T))

    R = T[:dim, :dim]

    if estimate_scale:
        # Eq. (41) and (42).
        scale = 1.0 / src_demean.var(axis=0).sum() * np.dot(S, d)
    else:
        scale = 1.0

    #T[:dim, dim] = dst_mean - np.dot(T[:dim, :dim], src_mean.T)
    T = dst_mean - scale * np.dot(R, src_mean.T)
    #T[:dim, :dim] *= scale

    return R, T, scale

The angle that I get for z is 179.1226 degrees which is obviously wrong. But if I use 2D landmarks, the angle for Z is 4 degrees which is the right value.

Is there something wrong with the 3D landmarks values?

[rakadambi]

I ran the same with 2D landmarks for src and dst and the angle returned is 0.77 degress which seems right. Is there some bug in my 3D landmarks code (I am just setting it to _3D). The Umeyama code seems to be fine.

[rakadambi]

One more data point. So, I gathered the 3D landmarks points but only used (x, y) and passed it to Umeyama algorithm and the Z rotation angle was right! When I use Z, it goes haywire!

Anybody can point me to a clue here?

[rakadambi]

Closing this. This is not a FA issue