Would it be complicated to add eyeblinking reference videos?

[G-force78]

Or adjust the keypoint detector to determine eyeblinking frequency?

Like in sadtalker , Ive struggled to find where in your codebase this could relate to

https://github.com/OpenTalker/SadTalker/blob/dev/src/facerender/modules/make_animation.py

current_directory = os.getcwd() print(current_directory)

    Set the source directory
    source_dir = os.path.dirname(os.path.abspath(__file__))

    ref_eyeblink = "C:\\Users\\yours\\SadTalker\\ref_eyeblink_video.mp4"
    print(ref_eyeblink)

    ref_pose = "C:\\Users\\yours\\SadTalker\\ref_pose_video.mp4"
    print(ref_pose)

    Code snippet (continued)
    if ref_eyeblink is not None:
        ref_eyeblink_videoname = os.path.splitext(os.path.split(ref_eyeblink)[-1])[0]
        ref_eyeblink_frame_dir = os.path.join(save_dir, ref_eyeblink_videoname)
        os.makedirs(ref_eyeblink_frame_dir, exist_ok=True)
        print('3DMM Extraction for the reference video providing eye blinking')
        ref_eyeblink_coeff_path, _, _ =  self.preprocess_model.generate(ref_eyeblink, ref_eyeblink_frame_dir)
    else:
        ref_eyeblink_coeff_path = None

    Code snippet (continued)
    if ref_pose is not None:
        if ref_pose == ref_eyeblink: 
            ref_pose_coeff_path = ref_eyeblink_coeff_path
        else:
            ref_pose_videoname = os.path.splitext(os.path.split(ref_pose)[-1])[0]
            ref_pose_frame_dir = os.path.join(save_dir, ref_pose_videoname)
            os.makedirs(ref_pose_frame_dir, exist_ok=True)
            print('3DMM Extraction for the reference video providing pose')
            ref_pose_coeff_path, _, _ =  self.preprocess_model.generate(ref_pose, ref_pose_frame_dir)
    else:
        ref_pose_coeff_path = None

[yuangan]

We have not considered eye blinking. Defining eye blinking like SadTalker in latent keypoints space is challenging. Finetuning our work with an eye-blinking prompt and related loss function may be a good solution for your need.