How to use our own dataset to get 3D Human Pose Estimation in Video with 2D keypoint json file

[jellyfish1456]

Hi there~ Thank you for sharing this great work!

My question is that how to use our own video to get 3D Human Pose Estimation with your pretrained model.

Thank you

[fabro66]

Hi~ Sorry that we have not yet provided the code for 2D keypoint generation.

I recommend you to use HRNet to generate 2D keypoints and save the keypoints format as like ./data/keypoints/baseball.json. Next, follow the 'inference in the wild' step to get your own 3D poses.

[jellyfish1456]

@fabro66 OK, I see. Thank you for your answer~

I just use Alphapose to generate 2D keypoints json file for video And I use that json file resulting in "openpose" format It also does not work either here.

So that now it only works for HRNet to generate 2D keypoints json file so far? Thank you so much~

[fabro66]

It may be that the saved keypoints are in a different format. Could you please post the 2D keypoints generated by Alphapose and let me see which link has bugs.

[jellyfish1456]

@fabro66 Here are two types of 2D Keypoint json files

1. This is the "openpose" format generating by Alphapose

2. This is the "default" format generating by Alphapose

[fabro66]

Hi~ According to the 2D keypoints file you provided. You can use the following load_json_openpose/alphapose to replace the load_json function in reconstrction.py. The function only supports load the keypoints estimated from images. But you can extend it to load keypoints from videos.

openpose

def load_json_openpose(kpts_file):
    with open(kpts_file, 'r') as fr:
        img_info = json.load(fr)

    num_person = len(img_info['people'])
    keypoints = np.zeros((num_person, 1, 18, 2), dtype=np.float32)  # (M, T, N, C)
    scores = np.zeros((num_person, 1, 18))  # (M, T, N)

    for i, human_info in enumerate(img_info['people']):
        kpts_scores = np.asarray(human_info['pose_keypoints_2d'], dtype=np.float32)
        kpts_scores = kpts_scores.reshape(18, 3)
        keypoints[num_person, 0] = kpts_scores[:, :2]
        scores[num_person, 0] = kpts_scores[:, 2:]

        return keypoints, scores

Alphapose

def load_json_alphapose(kpts_file):
    with open(kpts_file, 'r') as fr:
        img_info = json.load(fr)

    num_person = len(img_info)
    keypoints = np.zeros((num_person, 1, 17, 2), dtype=np.float32)  # (M, T, N, C)
    scores = np.zeros((num_person, 1, 17))  # (M, T, N)

    for i, human_info in enumerate(img_info):
        kpts_scores = np.asarray(human_info['keypoints'], dtype=np.float32)
        kpts_scores = kpts_scores.reshape(17, 3)
        keypoints[num_person, 0] = kpts_scores[:, :2]
        scores[num_person, 0] = kpts_scores[:, 2:]

    return keypoints, scores

[fabro66]

Hi~ Thank you for your interest in our work. In the future, I will release the newest version with improved GAST-Net for real-time multi-person 3D poses estimation. The project will contain human detection and tracking, 2D pose estimation, and 3D pose estimation methods.

[jellyfish1456]

@fabro66 Thank you so much~

[danial880]

Hi~ Thank you for your interest in our work. In the future, I will release the newest version with improved GAST-Net for real-time multi-person 3D poses estimation. The project will contain human detection and tracking, 2D pose estimation, and 3D pose estimation methods.

Yes, that's great. How long will it take?

[fabro66]

Hi~ Thank you for your interest in our work. In the future, I will release the newest version with improved GAST-Net for real-time multi-person 3D poses estimation. The project will contain human detection and tracking, 2D pose estimation, and 3D pose estimation methods.

Yes, that's great. How long will it take?

Hi~ We will release these codes next month. At the same time, a pre-trained casual model will be provided.