How to use custom pose 2D key-points?

[Harsh-Vavaiya]

@Shimingyi @kfiraberman I need "proper" guidance on, how can I use 2d points? because I have tried inserting it but the BVH file generated is not showing any movement while on your examples it's a working fine.

Note: "I've converted points into h36m format than given to the model."

[Shimingyi]

@Harsh-Vavaiya You can use the following code because your input is not a square. And also, refer to https://github.com/Shimingyi/MotioNet/issues/43, the current output is in camera space so you should rotate it by yourself, I am going to fix it with tricks in next commit (around 1st, May).

IMAGE_WIDTH = 480
IMAGE_HEIGHT = 270
...
    poses_2d = np.concatenate(pose_batch, axis=0)
    poses_2d[:, np.arange(0, poses_2d.shape[-1], 2)] /= (IMAGE_WIDTH*1)    
    poses_2d[:, np.arange(1, poses_2d.shape[-1], 2)] /= (IMAGE_HEIGHT*1)   

[Harsh-Vavaiya]

Ok, It didn't work.

I didn't mean the rotations in 3D, I've dealt with it. I meant pose of 3D structure. It should be 3D T-pose but it has lots of deformities.

[Shimingyi]

@Harsh-Vavaiya You mentioned you have converted points into h36m format, so have you compared your format with ours? There are 17 joints and ordered with following:

H36M_NAMES = [''] * 32
H36M_NAMES[0] = 'Hip'
H36M_NAMES[1] = 'RHip'
H36M_NAMES[2] = 'RKnee'
H36M_NAMES[3] = 'RFoot'
H36M_NAMES[6] = 'LHip'
H36M_NAMES[7] = 'LKnee'
H36M_NAMES[8] = 'LFoot'
H36M_NAMES[12] = 'Spine' 
H36M_NAMES[13] = 'Thorax'
H36M_NAMES[14] = 'Neck/Nose' 
H36M_NAMES[15] = 'Head'
H36M_NAMES[17] = 'LShoulder'
H36M_NAMES[18] = 'LElbow'
H36M_NAMES[19] = 'LWrist'
H36M_NAMES[25] = 'RShoulder'
H36M_NAMES[26] = 'RElbow'
H36M_NAMES[27] = 'RWrist'

[Harsh-Vavaiya]

Yes it is same

[Shimingyi]

I have no idea. Maybe you can just post the video here so I can share me processing steps.

[Harsh-Vavaiya]

ok sure.

https://drive.google.com/file/d/1D2pDpLFTwE79yFWyhUCJpdhxVoTEMkdr/view?usp=sharing

[Harsh-Vavaiya]

@Shimingyi Any update?

[Shimingyi]

@Harsh-Vavaiya No I am busy in few ddls, so pls wait me for a while. (1-2 days)

[Harsh-Vavaiya]

@Shimingyi are you available?

[Shimingyi]

@Harsh-Vavaiya I ran the code but got a bad result on your video: First frame looks like:

Wrong rotation problem: it should be clockwise but counterclockwise in our outputs.

We must recognize the limitation of the current method and training data. And until now, there is no planning to improve it in this version, and we need to wait for a new version that includes more features and robustness on camera view and input data format. But I hope the neural FK module can bring some inspiration to you.

[Harsh-Vavaiya]

@Shimingyi sure and thank you. also, can you tell me the procedure to train with a custom dataset? and have you updated the code on 1st May?

[anas-zafar]

@Harsh-Vavaiya were you able to train the model on your custom dataset?