search

facebookresearch / habitat-lab

A modular high-level library to train embodied AI agents across a variety of tasks and environments.
https://aihabitat.org/
MIT License
1.93k stars 483 forks source link

Connecting RGB sensor to a humanoid actual head #1735

Closed dantes-ai closed 8 months ago

dantes-ai commented 9 months ago

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: v0.3.0 Habitat-Sim: v0.3.0

❓ Questions and Help

I would like to train a humanoid robot to perform certain tasks (similar to a robot), whose main observation is an RGB sensor.

The implementation in kinematic_humanoid.py connects the sensor to the humanoid base link, which means that the sensor captures images from a viewport connected to the agent's root position.

How may I change it so that the sensor is attached to the humanoid head instead, allowing for a more realistic observation for the humanoid?

aclegg3 commented 9 months ago

@dantes-ai

You can follow the examples setup for robotic agents to synchronize camera transforms with a particular rigid link transform (in this case the humanoid model's head).

In manipulator.py the update function loops over all configured cameras to do this: https://github.com/facebookresearch/habitat-lab/blob/e9850f22f3659bef97a7f7aeabbbabc6561f4fce/habitat-lab/habitat/articulated_agents/manipulator.py#L139

Currently for humanoids, the camera is configured for attachment to the base link here: https://github.com/facebookresearch/habitat-lab/blob/e9850f22f3659bef97a7f7aeabbbabc6561f4fce/habitat-lab/habitat/articulated_agents/humanoids/kinematic_humanoid.py#L37

Instead, find the head link index and attach the camera there as is done for Spot's arm cameras here: https://github.com/facebookresearch/habitat-lab/blob/e9850f22f3659bef97a7f7aeabbbabc6561f4fce/habitat-lab/habitat/articulated_agents/robots/spot_robot.py#L114

dantes-ai commented 9 months ago

Thanks for your timely response!

Just to make sure I understand, you propose to subclass KinematicManipulator?

For example, in order to override _get_humanoid_params and change 'attached_link_id' to the newly desired link id.

xavierpuigf commented 9 months ago

@dantes-ai, yes you can subclass kinematic_humanoid for that. You can also directly edit kinematic_humanoid and add new cameras with different attached_link_id. For the same agent, you can have different cameras pointing to different links, see here: https://github.com/facebookresearch/habitat-lab/blob/main/habitat-lab/habitat/articulated_agents/robots/spot_robot.py#L114