something wrong with 'blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])'

[wjrzm]

I have configured the environment according to the installation process in README, but when I run ik_example_joints.py, I have the following error.

(py37)  ✘ wjrzm@DESKTOP-QJIK4OK  ~/human_body_prior/tutorials   master ±  /home/wjrzm/anaconda3/envs/pytorch1.7
.1/bin/python /home/wjrzm/human_body_prior/tutorials/ik_example_joints.py
No module named 'body_visualizer'
psbody.mesh based visualization could not be started. skipping ...
sample_amass keys:  ['poses', 'dmpls', 'labels', 'markers', 'mocap_framerate', 'betas', 'gender', 'vids', 'trans']
smplx (10475, 3, 486)
2023-03-17 15:25:20.281 | INFO     | human_body_prior.tools.model_loader:load_model:97 - Loaded model in eval mode with trained weights: ../support_data/dowloads/vposer_v2_05/snapshots/V02_05_epoch=13_val_loss=0.03.ckpt
[133 498 170 438]
smplx (10475, 3, 486)
Traceback (most recent call last):
  File "/home/wjrzm/human_body_prior/tutorials/ik_example_joints.py", line 105, in <module>
    ik_res = ik_engine(source_pts, target_pts)
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 292, in forward
    optimizer.step(lambda: closure(wts, free_vars))
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/optim/lbfgs.py", line 311, in step
    orig_loss = closure()
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 292, in <lambda>
    optimizer.step(lambda: closure(wts, free_vars))
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 142, in fit
    res = source_kpts_model(free_vars)
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/tutorials/ik_example_joints.py", line 50, in forward
    new_body = self.bm(**body_parms)
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/body_model.py", line 261, in forward
    dtype=self.dtype)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/lbs.py", line 209, in lbs
    v_shaped = v_template + blend_shapes(betas, shapedirs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/lbs.py", line 299, in blend_shapes
    blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/functional.py", line 342, in einsum
    return einsum(equation, *_operands)
  File "/home/wjrzm/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/functional.py", line 344, in einsum
    return _VF.einsum(equation, operands)  # type: ignore
RuntimeError: size of dimension does not match previous size, operand 1, dim 2

I searched for a lot of corresponding errors on the Internet. They tried to remind me of the BUG of the corresponding function of Pytorch, so I upgraded the version of Pytorch. The following is the error report using Pytorch1.12.1 and 1.13.1.

(py37)  wjrzm@DESKTOP-QJIK4OK  ~/human_body_prior/tutorials   master ±  /home/wjrzm/anaconda3/envs/py37/bin/pyt
hon /home/wjrzm/human_body_prior/tutorials/ik_example_joints.py
Platform does not define a GLUT font retrieval function
psbody.mesh based visualization could not be started. skipping ...
sample_amass keys:  ['poses', 'dmpls', 'labels', 'markers', 'mocap_framerate', 'betas', 'gender', 'vids', 'trans']
smplx (10475, 3, 486)
2023-03-17 15:31:24.031 | INFO     | human_body_prior.tools.model_loader:load_model:97 - Loaded model in eval mode with trained weights: ../support_data/dowloads/vposer_v2_05/snapshots/V02_05_epoch=13_val_loss=0.03.ckpt
[123 110  26 423]
smplx (10475, 3, 486)
Traceback (most recent call last):
  File "/home/wjrzm/human_body_prior/tutorials/ik_example_joints.py", line 105, in <module>
    ik_res = ik_engine(source_pts, target_pts)
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 292, in forward
    optimizer.step(lambda: closure(wts, free_vars))
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/optim/optimizer.py", line 113, in wrapper
    return func(*args, **kwargs)
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/optim/lbfgs.py", line 311, in step
    orig_loss = closure()
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 292, in <lambda>
    optimizer.step(lambda: closure(wts, free_vars))
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 142, in fit
    res = source_kpts_model(free_vars)
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/tutorials/ik_example_joints.py", line 50, in forward
    new_body = self.bm(**body_parms)
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/body_model.py", line 261, in forward
    dtype=self.dtype)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/lbs.py", line 209, in lbs
    v_shaped = v_template + blend_shapes(betas, shapedirs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/lbs.py", line 299, in blend_shapes
    blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/functional.py", line 358, in einsum
    return einsum(equation, *_operands)
  File "/home/wjrzm/anaconda3/envs/py37/lib/python3.7/site-packages/torch/functional.py", line 360, in einsum
    return _VF.einsum(equation, operands)  # type: ignore[attr-defined]
RuntimeError: einsum(): operands do not broadcast with remapped shapes [original->remapped]: [4, 16]->[4, 1, 1, 16] [10475, 3, 10]->[1, 10475, 3, 10]

(pytorch)  wjrzm@DESKTOP-QJIK4OK  ~/human_body_prior/tutorials   master ±  /home/wjrzm/anaconda3/envs/pytorch/b
in/python /home/wjrzm/human_body_prior/tutorials/ik_example_joints.py
Platform does not define a GLUT font retrieval function
psbody.mesh based visualization could not be started. skipping ...
sample_amass keys:  ['poses', 'dmpls', 'labels', 'markers', 'mocap_framerate', 'betas', 'gender', 'vids', 'trans']
smplx (10475, 3, 486)
2023-03-17 15:56:27.134 | INFO     | human_body_prior.tools.model_loader:load_model:97 - Loaded model in eval mode with trained weights: ../support_data/dowloads/vposer_v2_05/snapshots/V02_05_epoch=13_val_loss=0.03.ckpt
[460 154 324 100]
smplx (10475, 3, 486)
Traceback (most recent call last):
  File "/home/wjrzm/human_body_prior/tutorials/ik_example_joints.py", line 105, in <module>
    ik_res = ik_engine(source_pts, target_pts)
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 292, in forward
    optimizer.step(lambda: closure(wts, free_vars))
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
    out = func(*args, **kwargs)
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/optim/lbfgs.py", line 312, in step
    orig_loss = closure()
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 292, in <lambda>
    optimizer.step(lambda: closure(wts, free_vars))
  File "/home/wjrzm/human_body_prior/src/human_body_prior/models/ik_engine.py", line 142, in fit
    res = source_kpts_model(free_vars)
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/tutorials/ik_example_joints.py", line 50, in forward
    new_body = self.bm(**body_parms)
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/body_model.py", line 261, in forward
    dtype=self.dtype)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/lbs.py", line 209, in lbs
    v_shaped = v_template + blend_shapes(betas, shapedirs)
  File "/home/wjrzm/human_body_prior/src/human_body_prior/body_model/lbs.py", line 299, in blend_shapes
    blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/functional.py", line 373, in einsum
    return einsum(equation, *_operands)
  File "/home/wjrzm/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/functional.py", line 378, in einsum
    return _VF.einsum(equation, operands)  # type: ignore[attr-defined]
RuntimeError: einsum(): subscript l has size 10 for operand 1 which does not broadcast with previously seen size 16

I am using

Could you help me find a solution?

[oneThousand1000]

Hi, in ik_example_joints.py, you could try to set num_betas = 10 in IK_Engine like:

ik_engine = IK_Engine(vposer_expr_dir=vposer_expr_dir,
                      verbosity=2,
                      display_rc= (2, 2),
                      data_loss=data_loss,
                      stepwise_weights=stepwise_weights,
                      optimizer_args=optimizer_args,
                      num_betas = 10 # ADD
                      ).to(comp_device)

Hope this can help you.

[wjrzm]

Hi, in ik_example_joints.py, you could try to set num_betas = 10 in IK_Engine like:

ik_engine = IK_Engine(vposer_expr_dir=vposer_expr_dir,
                      verbosity=2,
                      display_rc= (2, 2),
                      data_loss=data_loss,
                      stepwise_weights=stepwise_weights,
                      optimizer_args=optimizer_args,
                      num_betas = 10 # ADD
                      ).to(comp_device)

Hope this can help you.

Thanks, this helped me a lot! !

[bring-nirachornkul]

Hi, in ik_example_joints.py, you could try to set num_betas = 10 in IK_Engine like:

ik_engine = IK_Engine(vposer_expr_dir=vposer_expr_dir,
                      verbosity=2,
                      display_rc= (2, 2),
                      data_loss=data_loss,
                      stepwise_weights=stepwise_weights,
                      optimizer_args=optimizer_args,
                      num_betas = 10 # ADD
                      ).to(comp_device)

Hope this can help you.

It's working. Thank you