search

chaneyddtt / Convolutional-Sequence-to-Sequence-Model-for-Human-Dynamics

Source code of paper "Convolutional Sequence to Sequence Model for Human Dynamics"
MIT License
86 stars 18 forks source link

Using 3D coordinates for joints as input. #5

Open zaverichintan opened 4 years ago

zaverichintan commented 4 years ago

I am wondering if this model works for 3D coordinates of joints as input?

I tried to train by changing the load_data in data_utils.py to load 32 joints data (96 dimensions) and also changing the dimensions in human_encoder.py.

I get a weird output as can be seen below: Video

Has someone else tried doing so successfully? Any help would be great. Thanks in advance.

chaneyddtt commented 4 years ago

Hi @zaverichintan, I did not try to input only one frame of 3d coordinates before, but i suspect that it would be very hard for the network to learn the motion dynamics with only one frame. You might have to do some modifications to the original network to get it work. It would be challenge I think because one frame can not provide enough information.

zaverichintan commented 4 years ago

Hello @chaneyddtt, I was wondering if you tried giving a sequence of frames (3d coordinates), as for me the network does not seem to learn the motion dynamics.

chaneyddtt commented 4 years ago

Can you give more details on how you preprocess the 3d coordinates data?

zaverichintan commented 4 years ago

There is no preprocessing done on the 3D coordinates. I am using mocap library and my code can be found on My commit at the forked repo

chaneyddtt commented 4 years ago

I think some preprocess step is needed in your case, such as set the root joint as the original point and normalize the data before feeding into the network.

zaverichintan commented 4 years ago

I perform normalization by setting the root joint at the origin and by rotating each 3D pose in such a way that it faces in the same direction, aligning with the y-axis. I trained the seq2seq model as well as trajectory dependency (https://github.com/wei-mao-2019/LearnTrajDep) with this data and it works for both models without any issues.

I am wondering if the kernel size or the order in which the different joints appear in the vector matter.

Thank you very much for your help!

chaneyddtt commented 4 years ago

This paper (https://github.com/wei-mao-2019/LearnTrajDep) also tests our model directly on 3d joint coordinates, can you get similar results as theirs?