Question about wrist camera.

[kaijieshi7]

Hi, the wrist camera used in rl environment is for problems like occlusion, and only use it to process images in real-life situations. Is that right?

[j96w]

Hi, you are correct, to give more details - besides occlusion and collision avoidance, the wrist camera in RL env is also utilized to determine the camera's position and orientation. This allows us to compute the relative 6D pose of the mega block within the camera's frame. The wrist camera images in RL env are not used. During real-world deployment, the relative pose can be directly estimated using RGB-D frames from the wrist camera.

[kaijieshi7]

For occlusion and collision avoidance, the virtual allegro hand model can handle this problem. But for the camera's position and orientation, I didn't find any code that uses the wrist camera, the "if" statement skips the using of wrist camera. Could you tell me where the code is useful for camera's position and orientation? thank you.

[j96w]

To clarify, the occlusion and collision avoidance we are discussing here is for the wrist camera's collision mesh instead of the hand model. But we later found the RL would keep using the wrist camera mesh to solve the task (use the camera to dig the blocks during the searching phase), so we ended up disabling the collision mesh of the camera and only using it as a visualization in this code.

The camera's pose and transformation are processed in the task environment file. Here is an example: https://github.com/sequential-dexterity/SeqDex/blob/master/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_grasp.py#L1182

[kaijieshi7]

Thank you very much.