Humanoid Picking Up Motion

[FanLu97]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: master

Habitat-Sim: master

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release versions of Habitat-Lab and Habitat-Sim? Your question may already be addressed in the latest versions. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Master branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-lab/

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/

Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

Hi, thanks for the good job. I have a question regarding the humanoid picking up motion, which is mentioned in your paper.

I run the provided sandbox tool and the humanoid walks as expected. However, when I want to pick up or drop an object, I do not see the motion of humanoid (the object directly moves to the hand of the humanoid or moves away). Do I missed anything?

Thanks in advance.

[xavierpuigf]

Hi, I am not sure if the way you are running the command, the picking motion is enabled or not, if you could provide some reproduction steps, it would be helpful. The part of the code that is in charge of calculcating the pick pose is here:

https://github.com/facebookresearch/habitat-lab/blob/main/habitat-hitl/habitat_hitl/environment/controllers/gui_controller.py#L492

Could you verify whether is_picking_reach_pose is set to some value when picking, and that your code gets there? If the tool gets there, it should be making a picking motion.

[ambervg]

Hi, I am encountering a similar problem with the humanoid picking up motion. To illustrate I have made a demo repo, which I will link here. I tried printing is_picking_reach_pose as such: self._gui_agent_ctrl._is_picking_reach_pos, but that keeps giving me None, also while picking and placing objects in the simulation.

It does update reach_pos in this line and feeds that into the GuiHumanoidController in this line. Next, _hint_reach_pos gets updated, but that's where I'm losing the thread.

Any tips on how to tackle this? If you need more info for clarification, please let me know!

[ambervg]

@FanLu97 Were you able to fix your issue in the end? @xavierpuigf Do you have any thoughts on this?

Just as clarification, for me the problem is occurring when I try to create a HITL app.

[xavierpuigf]

Taking a look at this!

[xavierpuigf]

Hi @ambervg, I don't have a local install to reproduce this now but can help with debugging, will try to update/clarify the code soon. The humanoid control actions work in 2 steps: set_act_hints where you set all relevant information to take actions and act where we use the hints to figure out how to update the agent joints. My suggestion would be to look at whether the information is correctly kept in the act function.

• Can you put a breakpoint here: https://github.com/facebookresearch/habitat-lab/blob/main/habitat-hitl/habitat_hitl/environment/controllers/gui_controller.py#L553 and verify whether hint_reach_pos is different than None?
• If it is None, even though you set hint_reach_pos, you should check if set_act_hints is being called anytime with reach_pos=None, before the act function https://github.com/facebookresearch/habitat-lab/blob/main/habitat-hitl/habitat_hitl/environment/controllers/gui_controller.py#L373. If so, we can debug from there.
• If it is not None ,then is_picking_reach_pos should be != None, I would put a breakpoint and check that you enter here: https://github.com/facebookresearch/habitat-lab/blob/main/habitat-hitl/habitat_hitl/environment/controllers/gui_controller.py#L558

Let me know!

[ambervg]

• When picking something, hint_reach_pos does indeed update to a vector such as Vector(-6.50458, 0.748031, -5.45327) instead of staying None.
• The is_picking_reach_pos is indeed != None and is being updated to the same vector as hint_reach_pos.

EDIT: This is the same for the example script pick_throw_vr.py. Just mentioning in case that is easier to work with / explain through.

[xavierpuigf]

Got it! It seems like things are being well set in the demo. I assume you are testing things on the keyboard?

I think that the current version of the demo doesn't do a full pick animation, but instead gets a single picking pose of the person reaching the object (while the space bar is pressed down). That position is set here: https://github.com/ambervg/habitat-hitl-app/blob/main/test.py#L441

The last thing to check is whether the humanoid avatar has the relevant poses to have a reaching pose: Could you check here that hand_data is not None? If it is not None I think that you should be seeing a reaching position whenever the space bar is pressed. if that is the case, the code should be adapted to have a reaching motion instead of directly the reaching pose. https://github.com/facebookresearch/habitat-lab/blob/main/habitat-lab/habitat/articulated_agent_controllers/humanoid_rearrange_controller.py#L622

[ambervg]

Correct, I'm not using the VR, just the keyboard.

I'm not seeing any picking pose in the simulation. hand_data is None, even when picking something by pressing space. I guess that corresponds well to the fact that there is no visible picking pose!

[xavierpuigf]

Interesting! Yeah if hand_data is None there will be no movement because ti requires the motion stored in hand_data. Could you try downloading the humanoid data from here? https://huggingface.co/datasets/ai-habitat/habitat_humanoids/tree/main

[ambervg]

I replaced the data/humanoids/humanoid_data on my computer with the humanoid data from huggingface, but the hand_data is still None.

[xavierpuigf]

Could you load the humanoid_data pkl file you are using (female_0_motion_data_smplx.pkl by default). And verify that it has the keys:

dict_keys(['walk_motion', 'stop_pose', 'left_hand', 'right_hand'])

You can check which file is being loaded here, it should be {male/female}_{0-9}_motion_data_smplx.pkl https://github.com/facebookresearch/habitat-lab/blob/main/habitat-lab/habitat/articulated_agent_controllers/humanoid_rearrange_controller.py#L57

[ambervg]

walk_pose_path: `data/humanoids/humanoid_data/walking_motion_processed_smplx.pkl`
dict_keys(['walk_motion', 'stop_pose'])  # So you are right! left_hand and right_hand are missing

The other files (e.g. data/humanoids/humanoid_data/neutral_0/neutral_0_motion_data_smplx.pkl) do have the left_hand and the right_hand keys. When linking that in the config.yaml I see a picking pose!! :partying_face:

https://github.com/facebookresearch/habitat-lab/assets/134403970/1871f151-2b7d-440b-8a29-7da2c42733b9

Some weird things still:

• After the picking pose, the humanoid is standing diagonal until resuming to walk.
• There is also a throwing motion, but this appears to be following the same movement every time, independent of the chosen location to throw the object to.
• Both the video and the simulation are quite choppy around the picking action. For the video, I did only do a screen recording, so maybe that is better when actually exporting a video from a sensor using habitat_sim.utils.viz_utils. Right now I only know how to export observations/video from the default sensors, which are all on the robot. If I could add sensors myself (e.g. a zoomed-out 1st person view), that would be awesome. I opened another issue on that earlier.