Reproducing the result

[PolarisXQ]

Thanks for your great work!

I am trying to reproduce the result you showed in the video. While I got almost the same result on Expi acro1_around-the-back1_cam20, there seems to be some different on Chi3D s03_Hug_7 which is also shown in the vedio.

This is the result I reproduce following the instructions in the readme

This is the result shown in your video

Seems like there is some error in the spatial placement, would you be so kind to tell me why this is happening?

I got this result by running

python run.py --cfg tcn_voxel_4_5_chi3d_multi_hum --data sample_data/chi3d_slahmr/chi3d_slahmr_s03_p1_phalpBox_all_slaInit_slaCam.pkl --data_name chi3d --name slahmr_override_loop2 --loops_uhc 2 --filter s03_Hug_7 

I also tried with

--loops_uhc 1 
--loops_uhc 2 
--loops_uhc 3 

but the spatial placement error still exists.

[nicolasugrinovic]

Hi, thanks for reporting this. This could be caused either by incorrect/different initial estimates, especially body shape, or some bug in this version of the code. I will try to reproduce it.

Can you tell, especially, from the first frames of the sequence, if there is a body shape difference between the humanoids body shape? They should be of different sizes.

[PolarisXQ]

Seems like the red one is shorter than the yellow one.

[nicolasugrinovic]

Can you please try with the following initial data?

1. Download the zip file from https://drive.google.com/uc?id=1le1Ca-KauPvbBX8TdrKeZhfFjzCniwQ1
2. place it under the ./sample_data/ folder
3. re-run the test code (changing the folder path if necessary)

If this does not solve the issue, it is some bug in the code and I will update the version, which should fix the problem. But please, first try this.

[PolarisXQ]

Your method works fine on s02_Hug_5, it's s03_Hug_7 that has the problem. I think you may want me to replace sample_data/chi3d_slahmr_s03_p1_phalpBox_all_slaInit_slaCam.pkl, but the data you provided is for s02.

[nicolasugrinovic]

That's right, sorry about that. Try this instead: https://drive.google.com/uc?id=1fbg2Js4Bu4wKLT5-NPIZoTYJk5pWZvzC

[PolarisXQ]

I re-run the test code with new sample_data.

run.py --cfg tcn_voxel_4_5_chi3d_multi_hum --data sample_data/chi3d_slahmr/chi3d_slahmr_s03_p1_phalpBox_all_slaInit_slaCam.pkl --data_name chi3d --name slahmr_override_loop2 --loops_uhc 2 --filter s03_Hug_7

The results haven't changed much.

[nicolasugrinovic]

I just updated the code to fix this. It should now reproduce the same results as shown in the video above. Please let me know if this works.

[PolarisXQ]

Cool! Now these humanoids can hug successfully :)

I'm curious what makes the estimated depths more accurate. Why the introduction of physical simulation can improve the performance of depth estimation?

[nicolasugrinovic]

Great!

To answer your question, the main insight (as mentioned in the paper), is that the physical simulation adds realistic constraints to the problem (which is generally underconstrained). The constraint in this scenario is that the simulated bodies cannot penetrate the ground floor or the other person thus the bodies cannot be placed freely but have to obey the laws of physics (as opposed to purely kinematic methods). The fact that the woman's arm cannot penetrate the man's body somehow serves as a "guidance" of where her body should be placed. Something similar happens with the ground. I hope this answers your question!

I am closing the issue, but if you have any more questions, please ask or open another issue!

[PolarisXQ]

Thank you for your reply, the method you proposed is very inspiring!