Question about new robot implementation

[ozgraslan]

Hello, Thank you for your amazing work. After reading the related paper and looking at the repo, I have two questions about adding new robots: 1) If I want to add a new robot to use within the environment for manual demonstration and reinforcement learning, are implementing mjcf and related environment python codes enough or should I also need to implement ik controllers using pybullet and include urdfs.

2) If pybullet and urdfs are needed, for what purposes these ik controllers and urdfs are used?

[edwhu]

Thanks for your kind words. We are adding additional robots (Panda, Jaco) for the next update. You can see the in-progress code on the dev branch at https://github.com/clvrai/furniture/tree/dev/env/models/robots. We plan to release the update in a few weeks. We will also update documentation on how to add robots.

To add a robot, you need at least the MJCF XML file to load it into MuJoCo. Next, you must implement some way to control the robot. We chose to use PyBullet framework because it has some robotic control methods. PyBullet requires an additional URDF file for the robot. The Pybullet code is wrapped in the ik_controller code.

Let me know if you have any more questions.

[ozgraslan]

Thanks for the answer. I will try to implement MJCF XML and ik_controller code. I will close the issue for now, I will reopen it if I have more questions.

[wangcongrobot]

Hi @ehu-ai. Thanks for the great project. I have created a new robot in mujoco and it works in the original robosuite. However, I don't know how to add the robot to the Unity project/scene. The original mujoco-unity plugin has a script that can import the robot, but it seems that the doorgym-unity plugin doesn't have the function: The MJCF import process is replaced with with a runtime loading mechanism triggerd through the TCP interface. So how could we deal with this? Thanks a lot. I use mujoco but not very familiar with Unity.

[edwhu]

Hello, Our environment automatically loads in the robot at runtime, just like doorgym-unity. It parses the Mujoco XML and loads in the robot mesh files for visualization at runtime.

So if you add a new robot, and want it to show up in unity, it must either have meshes or be composed of mujoco primitives. Please take a look at https://github.com/clvrai/furniture/tree/dev/env/models/assets/robots folder for example robot XMLs and their mesh files.

The unity code for loading in MuJoCo scenes is here https://github.com/clvrai/furniture/blob/dev/furniture-unity/Assets/Scripts/MJImport.cs

[wangcongrobot]

Hi @ehu-ai. Thanks a lot. I have understand what you said. The original mujoco import script can load the robot manually but the door-gym unity plugin can load it automatically as long as we put the robot files in correct folders. Now I can load my robot successfully.

What's more, when I use the demo_manul.py, it's doesn't work in the branch of dev. The master branch works fine. In normal, when we use keyboard to control the arm, Input action: will be printed. But in the dev branch, there is not any response when I use keyboard. I think maybe there are some bugs about the dev branch.

[edwhu]

Hello, This is because the Unity binary is different for master and dev since we update Unity functionality in dev. You will have to download the dev branch's Unity binary here if you want to try it https://drive.google.com/drive/u/1/folders/1-AobQkLqdBvJhW-AgUMF8i-fsmLSnqYW Alternatively, You can use the master branch for now since the robot configuration logic is the same between master and dev.

Thanks, Edward