search

mkocabas / VIBE

Official implementation of CVPR2020 paper "VIBE: Video Inference for Human Body Pose and Shape Estimation"
https://arxiv.org/abs/1912.05656
Other
2.85k stars 551 forks source link

[BUG] Fix "can not training with 2d dataset PennAction and PoseTrack" #256

Open winshot-thu opened 2 years ago

winshot-thu commented 2 years ago

Thanks @ZhangHedongTimesand @SatoshiYamazaki for the discussion: https://github.com/mkocabas/VIBE/issues/154

I fixed these bugs and summarized here.

PoseTrack: Replace
kp_2d = convert_kps(kp_2d, src=self.dataset_name, dst='spin')
with kp_2d = np.copy(self.db['joints2D'][start_index:end_index+1])

PennAction: When I visualized the keypoints in feature_extractor during preprocessing datasets, I found the first keypoint appeared is left ankle while "rankle" is the first one in def get_common_joint_names() function. All the keypoints is converse. When authors preprocessed the pennaction dataset, they converted penn format to common (converse) first in penn_action_utils.py. 

        perm_idxs = get_perm_idxs('pennaction', 'common')
        kp_2d[:, :, 0] = vid_dict['x']
        kp_2d[:, :, 1] = vid_dict['y']
        kp_2d[:, :, 2] = vid_dict['visibility']
        kp_2d = kp_2d[:, perm_idxs, :]

But kp_2d = convert_kps(kp_2d, src=self.dataset_name, dst='spin') in dataset_2d.py will regard penn keypoint format as pennaction and convert them to spin format.

So the following two stages will solve these problems.

  1. Replace function 'def get_common_joint_names()' in lib/data_utils/kp_utils.py with

    def get_common_joint_names():
return [
    "lankle",    
    "lknee",     
    "lhip",      
    "rhip",      
    "rknee",     
    "rankle",    
    "lwrist",    
    "lelbow",    
    "lshoulder", 
    "rshoulder", 
    "relbow",    
    "rwrist",    
    "neck",      
    "headtop",   
]

  2. Modify def get_single_item(self, index) in lib/dataset/dataset_2d.py as follow

         if self.dataset_name != 'posetrack':
        if self.dataset_name != 'pennaction':
            kp_2d = convert_kps(kp_2d, src=self.dataset_name, dst='spin')
        else:
            kp_2d = convert_kps(kp_2d, src='common', dst='spin')

These problems may be caused by the inconsistency between the index chosen by the author during preprocessing and the final code given. So the problems may only on downloaded feature datasets. If you preprocess dataset by yourself, these problems may not arise.