Shape of Model Output

[bryanbocao]

I am trying to understand the meaning of each dimension in: https://github.com/ZuoCX1996/Loose-Inertial-Poser/blob/37be233b1f2984a74571e77c3863cbdc42c23245/my_trainner.py#L353

I tried to find these information in both the code and the paper, it would be great to provide further clarity if I missed it.

The shapes of the two stages are as follows:

y_s1_hat.shape: torch.Size([256, 128, 33]), y_s2_hat: torch.Size([256, 128, 60])

Q1: y_s2_hat is the output of the model and it should be the input of the SMPL model, correct? Q2: Each dimension means (batch_size, length_of_the_interpolation_over_the_motion_sequence, ?). In this example, we have the output with batch_size of 256, length_of_the_interpolation_over_the_motion_sequence of 128 in the first two dimensions, what do 33 and 60 refer to? Q3: Suppose I am going to mask a certain IMU device with an index of the joint position, what is the best way to do that?

Thanks!

[bryanbocao]

From Equation (1) in the paper:

y_s1_hat should be

while y_s2_hat should be

The paper mentions 10 joints in the upper body:

But I am not sure how Jx3=10x3=30 can be translated to 33 or 60 in the code. It would be appreciated if you can provide more details. @ZuoCX1996 Thanks!

[ZuoCX1996]

Hi! @bryanbocao

Q1: Yes, y_s2_hat is the prediction of 10 upper body joints rotation in 6D representation. Q2: the 33 refer to 11 upper body endpoints' location, and the 60 is 10 upper body joints rotation in 6D representation. Q3: This typically requires referring to the joint index settings of the SMPL model. For example, if you want to mask the IMU worn on the right forearm, the corresponding SMPL joint index should be 19.

I have added live demo code to the project, and I hope it can help in understanding the model's data format!

[bryanbocao]

Hi @ZuoCX1996,

Thanks for answering most of my questions! I understand the full list can be found in https://github.com/Xinyu-Yi/TransPose/issues/69.

A follow-up of Q3: suppose I would like to mask an IMU device (on the right forearm for example) from the inputs to your model, while having the full upper body joints feeding into the SMPL model, Q3.1: I guess I should select the index i in the second dimension of amass_data[x_s1] and lip_data[x_s1] like amass_data[x_s1][:, i] for masking? https://github.com/ZuoCX1996/Loose-Inertial-Poser/blob/c382614b9a4cd2f3d3d2bc235391c58d654497d4/train_poser.py#L32-L34

train_poser.py - amass_data[x_s1].size(): torch.Size([7082503, 36])
train_poser.py - amass_data[joint_upper_body].size(): torch.Size([7082503, 33])
train_poser.py - amass_data[pose_all].size(): torch.Size([7082503, 60])

Q3.2: it where can I find the mapping of joint indices to arms? I tried to search https://github.com/ZuoCX1996/Loose-Inertial-Poser/blob/c382614b9a4cd2f3d3d2bc235391c58d654497d4/clothes_imu_syn.py#L7-L10 but I had difficulty in translating these numbers into the 4 upper body IMUs.

Thanks!

[ZuoCX1996]

Hi! @bryanbocao,

The amass_data['x_s1'] contains synthesized IMU data (4 IMUs) using the Amass Dataset, so it can be used directly as model‘s input.

IMU masking is only required in IMU data synthesize. In the process_amass function in Amass_preprocess.py, you can see how the joint mask (for rotation synthesis) and vertices mask (for acceleration synthesis) works.

The imu_axis you mentioned is for IMU data synthesis on the clothing mesh calculated by TailorNet. The corresponding code is in my modified TailorNet project. I did not include it in the LIP project due to potential copyright issues.