Question about landmarks and EMICA's reconstruction from processed MEAD dataset

[EvelynQIN]

I downloaded the processed MEAD dataset using download_processed_mead.sh you provided. However, it seems that the landmarks from the processed/landmarks_original/.../landmarks.pkl do not align with the emica's reconstruction after flame lbs and orthogonal projection using the predicted 'cam', 'shape', 'exp' and 'pose'. Could you kindly elaborate on what the input images (how to warp it) of the EMICA is?

The first image is the processed 478 landmarks (from processed/landmarks_original/.../landmarks.pkl) drawn on the warped image using the landmarks_original.pkl. The second image is the projected 2d mediapipe lmks onto the same image using the reconstruction from the processed/reconstructions/.../shape_pose_cam.hdf5.

[radekd91]

We did not release the cropped videos since Mead does not belong to us. The landmarks are with respect to those. The script process_mead.py can extract those. For that you first need to download the original mead dataset and resample it to 25fps. The script can then run face detection on each frame and create a video with an aligned face.

On Sun, May 5, 2024, 10:33 EvelynQIN @.***> wrote:

I downloaded the processed MEAD dataset using download_processed_mead.sh you provided. However, it seems that the landmarks from the processed/landmarks_original/.../landmarks.pkl does not align with the emica's reconstruction after flame lbs and orthogonal projection using the predicted 'cam', 'shape', 'exp' and 'pose'. Could you kindly elaborate on what is the input image (how to warp it) of the EMICA?

— Reply to this email directly, view it on GitHub https://github.com/radekd91/inferno/issues/17, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7MKHOPUONC7TOL2OVRAFDZAZUO7AVCNFSM6AAAAABHH4PBY2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TSNRUGI2TONI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[EvelynQIN]

We did not release the cropped videos since Mead does not belong to us. The landmarks are with respect to those. The script process_mead.py can extract those. For that you first need to download the original mead dataset and resample it to 25fps. The script can then run face detection on each frame and create a video with an aligned face. … On Sun, May 5, 2024, 10:33 EvelynQIN @.> wrote: I downloaded the processed MEAD dataset using download_processed_mead.sh you provided. However, it seems that the landmarks from the processed/landmarks_original/.../landmarks.pkl does not align with the emica's reconstruction after flame lbs and orthogonal projection using the predicted 'cam', 'shape', 'exp' and 'pose'. Could you kindly elaborate on what is the input image (how to warp it) of the EMICA? — Reply to this email directly, view it on GitHub <#17>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7MKHOPUONC7TOL2OVRAFDZAZUO7AVCNFSM6AAAAABHH4PBY2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TSNRUGI2TONI . You are receiving this because you are subscribed to this thread.Message ID: @.>

Thank you so much for the timely reply. I have already downloaded the original MEAD video and resampled it to 25 fps following the script. The problem is the misalignment of the detected landmarks and the EMICA_flame_2020 reconstruction from the pre-processed MEAD dataset you provided. I warped the image following "process_mead.py" which is aligned with the landmarks.pkl. But the projected landmarks using the EMICA_flame_2020's prediction are shifted from the detected ones. Maybe these are not aligned since EMOTE does not use 2D supervision and I would need to rerun the process_mead.py again to get aligned landmarks and EMICA-reconstructions?

I test the alignment using the following code.

# read the EMICA_MEAD_flame2020 reconstruction
filename = 'dataset/mead_25fps/processed/reconstructions/EMICA-MEAD_flame2020/M003/front/angry/level_1/001/shape_pose_cam.hdf5'
data_dict = {}
with h5py.File(filename, "r") as f:
    for k in f.keys():
        print(f"shape of {k} : {f[k].shape}")
        data_dict[k] = torch.from_numpy(f[k][:])

flame = FLAME_mediapipe(cfg)

# get the reprojected 2d lmk of the first frame
frame_id = 0
shape = data_dict['shape'][:,frame_id]
exp = data_dict['exp'][:,frame_id]
pose = data_dict['pose'][:,frame_id]
cam = data_dict['cam'][:,frame_id]
verts, lmk_3d = flame(shape, exp, pose)

lmk2d_pred = batch_orth_proj(lmk_3d, cam)[:, :, :2]
lmk2d_pred[:, :, 1:] = -lmk2d_pred[:, :, 1:]

lmk2d_pred = lmk2d_pred[0] * 112 + 112

# test lmk alignment with the detected ones
lmk_aligned_path = 'dataset/mead_25fps/processed/landmarks_original/mediapipe/M003/front/angry/level_1/001'
with open(os.path.join(lmk_aligned_path, 'landmarks.pkl'), 'rb') as f:
    landmark_478 = pickle.load(f)
landmark_478 = np.asarray(landmark_478).squeeze(1)
lmk2d_detected = landmark_478[frame_id]

# draw the landmarks
blank_img = np.zeros((224, 224, 3)).astype(np.uint8)
draw_lmk_on_image(blank_img, lmk2d_delected, lmk2d_pred)

Here is the test result. It seems setting bb_center_shift_y=0 (instead of -1) can make the reconstruction align with the warpe image.