How to organize the 2d pose data from Detectron + CPN for run.py in wild video?

[yunfanLu]

Thanks for your attention.

This the detail for test in wild.

1. I have using this method get the detectron result. like this

this have two useful info pose and det, the pose is a ndarray , which shape is (17, 4), (x, y, logit score before softmax, probability score after softmax)

2. Then I try to transform the data to VideoPose3D Pose data。

def load_detr_res(detr_res):
    """
    Note: the open shuld be 'rb'
    Load the detectron keypoint detection result.
    for keypoint it has 4 value, (x, y, logit score before softmax, probability score after softmax),
    so you just need to take the indices 0, 1, and 3
    :param detr_res: dict {det, seg, pose}
    :return:
    """
    with open(detr_res, 'rb') as f:
        d = pickle.load(f)
    return d

def detr_to_h36m(detr, prefix='frames_1s'):
    frame_len = len(detr.keys())
    h36m_list = [0 for i in range(frame_len)]
    for frame_item in detr.keys():
        val = detr[frame_item]
        pose = val['pose'][1][0]

        pose_xy = [[0, 0]for i in range(17)]
        for i in range(2):
            for j in range(17):
                pose_xy[j][i] = pose[i][j]

        frame_ind_str = frame_item[-7:-4]
        frame_ind_int = int(frame_ind_str)

        h36m_list[frame_ind_int] = pose_xy

    action = {'S1' : {
        'Walking 1': [np.array(h36m_list)]
    }}
    return action # shape=(frame_num, 17, 2)

def save_as_npz(frame_ns_h36m, name='frame_1s_h36m'):
    np.savez(f'{name}.npz',positions_2d=frame_ns_h36m, positions_3d={})

def dict_to_h36m_npz(dict_path, out_path='frame_0p05s_h36m'):
    detr = load_detr_res(dict_path)
    frame_1s_h36m = detr_to_h36m(detr)
    save_as_npz(frame_1s_h36m, name=out_path)

The detr_path is the result of detectron.

3. Then I get the frame_0p05s_h36m.npz

p2d = frame_0p005_p2d['positions_2d'].item()

p3d = frame_0p005_p2d['positions_3d'].item()

This struct is same as data/data_2d_h36m_cpn_ft_h36m_dbb.npz but has only S1 and 'Walking 1' key.

4. Then I change in run.py change this line https://github.com/facebookresearch/VideoPose3D/blob/cf94b42e1edf144f8eff2da0f43be737225e43bd/run.py#L61 ,

keypoints = np.load('data/data_2d_' + args.dataset + '_' + args.keypoints + '.npz')

keypoints_symmetry = keypoints['metadata'].item()['keypoints_symmetry']
kps_left, kps_right = list(keypoints_symmetry[0]), list(keypoints_symmetry[1])

keypoints = np.load('data/wild/frame_0p05s/frame_0p05s_h36m.npz')

5, This method is fail.

[MartinPlantinga]

Hi @yunfanLu, have you managed to fix this?