get_xyz from model output

[qiqiApink]

I set the weight lambda_rcxyz>0 to add the reconstruction loss of xyz positions when I run the trainning code of HumanML3D. But I got the following error:

creating data loader...
Reading ././dataset/humanml_opt.txt
Loading dataset t2m ...
100%|██████████| 23382/23382 [11:59<00:00, 32.51it/s]
Pointer Pointing at 0
creating model and diffusion...
TRANS_ENC init
EMBED TEXT
Loading CLIP...
Total params: 17.88M
Training...
Starting epoch 0
  0%|          | 0/383 [00:03<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/petrelfs/zhangyaqi/anaconda3/envs/t2m-gpt/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/mnt/petrelfs/zhangyaqi/anaconda3/envs/t2m-gpt/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/train/train_mdm.py", line 49, in <module>
    main()
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/train/train_mdm.py", line 45, in main
    TrainLoop(args, train_platform, model, diffusion, data).run_loop()
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/train/training_loop.py", line 138, in run_loop
    self.run_step(motion, cond)
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/train/training_loop.py", line 207, in run_step
    self.forward_backward(batch, cond)
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/train/training_loop.py", line 233, in forward_backward
    losses = compute_losses()
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/diffusion/respace.py", line 97, in training_losses
    return super().training_losses(self._wrap_model(model), *args, **kwargs)
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/diffusion/gaussian_diffusion.py", line 1309, in training_losses
    target_xyz = get_xyz(target)  # [bs, nvertices(vertices)/njoints(smpl), 3, nframes]
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/diffusion/gaussian_diffusion.py", line 1244, in <lambda>
    get_xyz = lambda sample: enc.rot2xyz(sample, mask=None, pose_rep=enc.pose_rep, translation=enc.translation,
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/model/rotation2xyz.py", line 52, in __call__
    rotations = geometry.rotation_6d_to_matrix(x_rotations[mask])
  File "/mnt/lustre/zhangyaqi/Motion-Generation/motion-diffusion-model/utils/rotation_conversions.py", line 533, in rotation_6d_to_matrix
    b3 = torch.cross(b1, b2, dim=-1)
RuntimeError: inconsistent tensors sizes input: [12544, 262, 1] other: [12544, 262, 0]

Can you help me?

[GuyTevet]

rcxyz loss, and all other geo losses, are supported only for the action2motion dataset. For the text2motion dataset, the geometric data is regulated using the data representation instead of explicit losses.

[GuyTevet]

More info here: https://github.com/GuyTevet/motion-diffusion-model/issues/19

[qiqiApink]

So the "pose_rep" which is fixed to "rot6d" https://github.com/GuyTevet/motion-diffusion-model/blob/3294a7fb6a74054074c1f948d271f0641765f4ae/utils/model_util.py#L45 is useless to text2motion setting?

[GuyTevet]

Yup, in this context we only care about data_rep, and data_rep is hardcoded for each dataset, you can not change it.