AddBiomechanics results do not superimpose with the SMPL sequence

[MarilynKeller]

I followed the steps you mentioned using NimblePhysics. However, why are the results positioned side by side instead of overlapping like in the image below? And why are some parts not showing the same pose? Also, why aren't the jumps landing on the ground?

This was generated with the command from the readme:

python3 smpl2ab/show_ab_results.py --osim_path=/home/hykong99/SKEL/SMPL2AddBiomechanics/addbio_res_folder/01/Models/match_markers_but_ignore_physics.osim --mot_path=/home/hykong99/SKEL/SMPL2AddBiomechanics/addbio_res_folder/01/IK/01_01_poses_segment_0_ik.mot --smpl_motion_path=/home/hykong99/SKEL/SMPL2AddBiomechanics/models/bsm/sample_motion/01/01_01_poses.npz

Originally posted by @hykong99 in https://github.com/MarilynKeller/SMPL2AddBiomechanics/issues/5#issuecomment-2179647965

[MarilynKeller]

The visualization works for me but I'll try to regenerate with AddBiomechanics and see.

In the meanwhile, the video would be more helpful to see what is going on. You can export it from the viewer by cliching on file > save as video > export. The terminal will print where the video has been saved and you can drop it here.

[MarilynKeller]

I regenerated the sequence and it looks like this.

The output skeleton superimposes with SMPL, but since the skeleton sequence is segmented in parts and you are only visualizing the first section, it stops before the SMPL sequence.

https://github.com/user-attachments/assets/2cf1aa2f-3f30-4027-8358-9a0b79c43587

You can find the other segments in the IK folder output by Addbiomechanics. You would have to concatenate the .mot files if you want to visualize the whole sequence:

[hykong99]

I have confirmed that merging the .mot files allows for visualization of the whole sequence. However, when using AddBiomechanics, I keep encountering the following error. The data I used is from your GitHub.

[MarilynKeller]

Please click on Show Processing Log and paste the content here.

[MarilynKeller]

It might be this error I just updated bsm.osim to fix this, please redownload it from the SKEL webpage and try the AddBiomechanics processing with this updated bsm.osim.

[hykong99]

I retry AddBiomechanics, but Same error occur. (There is gap between SMPL sequence and osim file)

[MarilynKeller]

1) How could you re-generate the results if AddBiomechanics crashes ? 2) I can not diagnode the AddBiomechanics problem without seeing the full log 3) I can not diagnose the superimposition without seing the video

Please download the new .bsm from the website (I updated it today), re-run Addbiomechanics, visualize these new result, and post the full result video on this thread as indicated earlier. Also please try to keep the issue in this thread, it is hard to follow otherwise.

[hykong99]

1. I just wait for the process to finish.

2. This is the full log and result screenshot. log.txt

3. This is the video https://github.com/user-attachments/assets/36df4f2b-d4d4-4794-a896-e7e06c515b70

And I would like to hear your answer to this question. [https://github.com/MarilynKeller/SMPL2AddBiomechanics/issues/1]

[MarilynKeller]

2. So it seems like AddBiomechanics does not throw an error anymore, right?

3. Thank you, there is an option in the visualization scipt to offset the two meshes , maybe it is on. Did you modify smpl2ab/show_ab_results.py? In the viewerEditorbox, if you unrollSMPL motionandOpenSim skeleton motion`, are the translations different than zero?

   

[hykong99]

I don't modify smpl2ab/show_ab_results.py. And aitviewer editor box's value is all zero. Also, there is no neck(cervical).

[MarilynKeller]

Thanks for the captures. This is strange, maybe try a git pull of smpl2addbio and run againsmpl2ab/show_ab_results.py ? I'll check what happend to the cervicals

[hykong99]

I already run the newest code.

[hykong99]

What SMPL sequence data did you use? I use other SMPL sequence dataset (in AMASS), but I still encounter same error. (There is gap between SMPL sequence and osim file)

https://github.com/user-attachments/assets/da63fb5f-2a10-471a-a539-f9251f76e8d0 https://github.com/user-attachments/assets/620a64c1-d695-48ac-96dd-e2508a303e92 https://github.com/user-attachments/assets/f36675ed-6414-4a1a-96ad-2ab2c22c675f https://github.com/user-attachments/assets/01536e1a-ed69-44d3-9822-a9a23e70f839

[hykong99]

And I use Geometry of AddBiomechanics version 24.05.20 Because I encounter this error. (refer to https://github.com/MarilynKeller/SMPL2AddBiomechanics/issues/1)

[MarilynKeller]

The warnings for the mesh usually are not a problem (I just posted a clarification on the post you linked). I suspect it is a visualization issue but I'm surprised as I don't have it one my side. Could you send me the result folder you downloaded from addbiomechanics for the example sequence (01_01)? Then I can test the visualization.

[MarilynKeller]

I realize on your videos that the markers set on SMPL (green spheres) seems to be all over the place. Did you modify data/bsm_markers.yaml ? If yes please also provide the custom marker set you used.

[hykong99]

I don't upload .npz file. I used SMPL sequence from your GitHub : skel_models_v1.1/sample_motion/01_01_poses.npz If you want to recieve my data, please tell me your email.

And I use marker set from your GinHub. But I think bsm_marker.yaml has some error.

I run show_marker.py using below command. "bsm_markers_smplx.yaml" does not cause a displacement error, but "bsm_markers.yaml" does have a displacement error.

python3 smpl2ab/show_markers.py --osim_path=/home/hykong99/SKEL/SMPL2AddBiomechanics/models/bsm/bsm.osim --smpl_markers_path=/home/hykong99/SKEL/SMPL2AddBiomechanics/smpl2ab/data/bsm_markers_smplx.yaml

"bsm_markers.yaml" Result

"bsm_markers_smplx.yaml" Result

[hykong99]

I change 9line in SMPL2Biomechanics/smpl2ab/show_ab_results.py

parser.add_argument('--smpl_markers_path', type=str, default=cg.bsm_markers_on_smpl_path, help='Path to SMPL markers') -> parser.add_argument('--smpl_markers_path', type=str, default=cg.bsm_markers_on_smplx_path, help='Path to SMPL markers')

So, I get correct video. https://github.com/user-attachments/assets/e7a3eb41-6771-45e0-8891-2e63e0d4c601

[MarilynKeller]

I understand now, the smplx support was a pull request I merged but did not pull and test it locally. Apology for this.

To clarify, you can generate the markers from two different models, either SMPL or SMPLX. And both model meshes have different vertex numbers and indices.

If you use SMPLH sequences from AMASS you should use SMPL. If you use SMPLX sequences, used SMPLX.

Markers on SMPL

python smpl2ab/show_markers.py --body_model smpl --marker_set_path smpl2ab/data/bsm_markers.yaml

Markers on SMPLX

python smpl2ab/show_markers.py --body_model smplx --marker_set_path smpl2ab/data/bsm_markers_smplx.yaml

Note the difference of topology, SMPLX has more vertices on the head.

I just pushed this scripts update but I still need to check the other scripts.

[MarilynKeller]

So in a nutshell, the display offset was due to the code loading SMPLX instead of SMPL for visualization.

Now it should be fixed if you pull and run again the visualization script :

python smpl2ab/show_ab_results.py \
--osim_path=/path/to/01/tmp/tmp7uh2nzzn/01/Models/match_markers_but_ignore_physics.osim  \
--mot_path=/path/to/01/tmp/tmp7uh2nzzn/01/IK/01_01_poses_segment_0_ik.mot \
 --smpl_motion_path=models/bsm/sample_motion/01/01_01_poses.npz 

If I append --body_model=smplx, I can reproduce what you were observing. The sequence is loaded as if the parameters were smplx parameters, which give different body shape, pose and markers than the smpl sequence used to fit the skeleton.

[MarilynKeller]

Also I added a --z_up argument to display the sequence upright. Please pull aitviewer-skel and smpl2addbio

[hykong99]

I encountered other errors. I think you need to add smpl_models:"path/to/me" in aitvconfig.yaml (in first line) When I don't add smpl_models:"path/to/me", it doesn't still work. (same result as above) But, I add smpl_models:"path/to/me". But it occurs error. And I wonder why I set --body_model=smpl, but it looks for the model in the smplx folder.

[MarilynKeller]

Given the error, it might be an issue with your smplh. The smplh pickles are missing a key (hand_components). I remember there is a script to run to merge Mano into smplh. Did you do that?

https://github.com/vchoutas/smplx/tree/main/tools#merging-smpl-h-and-mano-parameters

[MarilynKeller]

And for this question

And I wonder why I set --body_model=smpl, but it looks for the model in the smplx folder.

It is normal. The Amass sequences are SMPL-H sequences (SMPL with MANO hands). These are loaded using the smplx library. This library lets you load the smpl, smplh and smplx models and expect the models dictionary to be stored in the smplx folder (I see your smplh folder is in the smpl folder, not smplx).

[hykong99]

I want to use smpl. When I set to --body_model=smpl, the path reached smplh.

[MarilynKeller]

Then modify this line https://github.com/MarilynKeller/SMPL2AddBiomechanics/blob/4f107c08df95ac390ca5cacd88de86b295b4412d/smpl2ab/show_ab_results.py#L35 to body_model = 'smpl'

But then you'll need a smpl sequence. The provided example is a smplh sequence.

[hykong99]

So, how can I use smplh? The ReadME only describes how to install smpl and smplx.

[MarilynKeller]

Instructions to setup smplh are here https://github.com/vchoutas/smplx/blob/main/tools/README.md

[hykong99]

Thanks!!! I solve this error!

But I encounter other error. I use your code(show_marker.py). The body model result is correct. However, The OpenSim model is not correct.

1. There is a gap between OpenSim model and OpenSim marker.
2. OpenSim model is rotated