Reference works for 3D pose estimation problems

ChrisWu1997 / 2D-Motion-Retargeting

PyTorch implementation for our paper Learning Character-Agnostic Motion for Motion Retargeting in 2D, SIGGRAPH 2019

https://motionretargeting2d.github.io

MIT License

443 stars 86 forks source link

Reference works for 3D pose estimation problems #30

Closed RahhulDd closed 3 years ago

RahhulDd commented 3 years ago

Hi @ChrisWu1997, I have reworked your project to apply on to my dataset and it seems to bit learning things quite good. I am curious about how to perform same for 3D setting or atleast some other problem like motion synthesis. I am new to this field and quite interested to learn from more projects like yours. Can you refer to me a review paper or some github repo as a starting point that has performed in 3D setting. Thank you.

ChrisWu1997 commented 3 years ago

Hi @RahhulDd, Thanks for your interest. I'm not that active in this field nowadays, but I would recommend you the following works:

Neural Kinematic Networks for Unsupervised Motion Retargetting (motion retargeting in 3D)
MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency (3d pose reconstruction in joint-angle representation)
Skeleton-Aware Networks for Deep Motion Retargeting (general-purposed framework to represent skeleton motion, with motion retargeting in 3d as an application)

Plus, our work can also be extended to 3d motion retargeting by changing the data. If you have a more specific topic in mind, then maybe I can list more works.

RahhulDd commented 3 years ago

@ChrisWu1997 Thank you for your help. I have some doubts regarding 3D preprocessing pipeline. I was trying to normalize my data by killing it rotations around z-axis and remove global displacement. I achieved global displacement one but wasn't able to remove rotation around z-axis. Any idea how to remove it? Also when I converted the global to local coordinate system, the scales of the limbs were still intact which caused disfigured skeletons. I tried normalising limbs by dividing each limb by its own magnitude but didn't work. Any idea on this one too?

ChrisWu1997 commented 3 years ago

I was trying to normalize my data by killing it rotations around z-axis and remove global displacement. I achieved global displacement one but wasn't able to remove rotation around z-axis. Any idea how to remove it?

By z-axis, you refer to the character's global rotation or the local rotation of each bone? For both cases, I don't understand why should it be removed. Can you explain a bit more?

Also when I converted the global to local coordinate system, the scales of the limbs were still intact which caused disfigured skeletons. I tried normalising limbs by dividing each limb by its own magnitude but didn't work. Any idea on this one too?

I didn't fully get the point. How do you define the local coordinate system? Is it the relative transformation of each bone to its parent bone? If so, why should the scale of the limb to be changed?

RahhulDd commented 3 years ago

Hi @ChrisWu1997 , By removing rotations I was referring to the attached part of a research paper. Please let me know anything about it.

RahhulDd commented 3 years ago

@ChrisWu1997 The local coordinate system is same as yours by keeping it at pelvis point. The image below is the result I am getting. First I tried to center the position of coordinate system at pelvis point then I removed the global velocity by subtracting pelvis velocity from all joints and itself. The projection is after this but I wanted to see why the scales of limbs are so weird. For example look at the forearms which are extraordinarily longer than upper arms.

ChrisWu1997 commented 3 years ago

By removing rotations I was referring to the attached part of a research paper. Please let me know anything about it.

Looks like they indeed remove the global rotation around z-axis. This would make their motion trajectory always facing front. I'm not able tell why they did this based on this paragraph.

The image below is the result I am getting. First I tried to center the position of coordinate system at pelvis point then I removed the global velocity by subtracting pelvis velocity from all joints and itself.

I think it's just a visualization problem. The three axises are not set equal: the vertical axis is of range (-0.6, 0.4) and the ground axis is of range (-0.05, 0.15). I think you need to manually set the axis range to be equal, otherwise matplotlib will by default set the range by the maximum value along each axis.