[Question] Seeking Advice: Training Nao Robot with RL using IsaacLab – Mimic Joints, Textures, and Physics Issues

[louislelay]

Hello everyone,

I'm working on a project to train the Nao robot using a rl agent. I've made some progress but encountered obstacles and would greatly appreciate your insights to ensure I'm on the right track.

Steps I Followed

1. Asset Import

   • I followed the Importing a new asset tutorial.
   • Used the URDF from this repository and meshes from this link.
   • Removed gazebo and transmission tags as instructed, and the modified URDF is attached at the end.
   • Converted the URDF to USD, I encountered the following error but when I followed this tutorial, I had also this error so I'm not really worried for this:

     Runtime Error: Failed to open layer @.../instanceable_meshes.usd@

     Despite this, the model rendered correctly in Isaac Sim.

3. Environment Setup

   • Created an RL environment inspired by this tutorial.
   • Adapted the humanoid direct environment and took inspiration from the Franka-Cabinet example to import my custom asset. The environment files are attached in a zip.

4. Testing with a Dummy Agent

   • Ran the following command to test the setup:

     ./isaaclab.sh -p source/standalone/environments/zero_agent.py --task Isaac-Nao-Direct-v0 --num_envs 32

   • The robots appeared stuck in the ground and generated these types of errors:

     [Error] [omni.physx.plugin] Usd Physics: the revolute joint at prim /World/envs/env_0/Nao/LFinger22_link/LFinger23 does need a finite limit set to be used by the mimic joint feature.
     [Error] [omni.hydra] [UsdToMdl] Prim '/__Prototype_38/Looks/material_RThumb2UV/Shader' parameter 'diffuse_texture': References an asset that can not be found: './Props/materials/textureNAO.png'

Issues

1. Mimic Joint Error

   [Error] [omni.physx.plugin] Usd Physics: the revolute joint at prim /World/envs/env_0/Nao/LFinger22_link/LFinger23 does need a finite limit set to be used by the mimic joint feature.

   • This error occurred for each finger joint. I tried changing the joints to revolute and setting limits but had no success, it just changed my errors to:

     [Error] [omni.physx.plugin] Usd Physics: PhysxMimicJointAPI at /World/envs/env_0/Nao/l_wrist/LFinger11 must have exactly 1 "referenceJoint" relationship defined.

   • Removing the mimic tags resolved the issue, but I’m open to better solutions.

2. Texture Error

   [Error] [omni.hydra] [UsdToMdl] Prim '/__Prototype_38/Looks/material_RThumb2UV/Shader' parameter 'diffuse_texture': References an asset that can not be found: './Props/materials/textureNAO.png'

   • The textures initially appeared correctly after the URDF conversion but now render as white. Any ideas why this happens?

3. Robot Stuck

   • Modifying the initial pose didn’t help; the robot seems stuck and doesn’t respond to gravity as expected.

Setup

• Isaac Sim : 4.2
• Ubuntu 22.04

Questions

1. Did I miss any critical steps or make mistakes in the process?
2. Are there better ways to handle mimic tags?
3. How can I resolve the issue of the robot being stuck in the ground or floating?
4. (Optional) Any suggestions to fix the texture issue?

Thanks in advance for your help!

---

URDF modified : with mimic tags : nao_description_v1.zip without : nao_description_v2.zip

Environment files : nao.zip

[RandomOakForest]

Thanks for posting this. There are several items to discuss here. To start, have you tried following this example and visualize the collision meshes?

I will move your post into our Discussions to better address this and follow up on your example.