[Unity plugin] Problems with object interpenetration and tangential frictions

[Bargez908]

Hi,

I'm a researcher and I'm trying to use MuJoCo (3.1.4) with the Unity plugin (Unity version 2023.2.20f1) to create a simulation environment to use with a vr headset to grasp objects with the shadow hand from MuJoCo managerie.

During test i noticed two major problems:

1) Both the shadow hand and a cylinder created inside Unity as Mj Geom interpenetrate with each other/the enviroment.

2) I am not able with the shadow hand to apply any tangential friction but only normal ones (this is evident when i tried to slide the cylinder by swiping the hand above it, but also in another test when i tried to grasp a standing cylinder and i was not able to lift it)

I tried various friction settings but none seem to work (using only MuJoCo's bin/simulate everything works fine).

https://github.com/google-deepmind/mujoco/assets/127102416/01e01031-ac3c-4db8-a1dc-fc5c35ab132d

[aadhithya14]

Hi @Bargez908 , I would like to how you are you sending actions to the endeffector of the robot in unity. Is it through ZMQ sockets? I wanted to know if you are just using unity for rendering or your simulation is also unity itself?

[Bargez908]

Hi, currently we are controlling the position and rotation of the hand with a Mj Mocap (vive tracker) with the "tracked pose diver" as parent for the hand. Currently we are using Unity with the MuJoCo plugin where all the objects are decleared as MuJoCo objects (the hand too).

[Balint-H]

@Bargez908 This looks like you need to configure the MjGeoms in your scene to use higher condim (if its at 1, only normal forces are generated). Similarly by tweaking solimp and solref you can make geoms more resistant to penetration.

Relevant documentation: https://mujoco.readthedocs.io/en/stable/XMLreference.html#body-geom-condim

If that's not what causing it let me know.

Also, to confirm, you control a mocap body, then use a weld constraint to softly move the arm's base there, right?

[Bargez908]

Changing condim solved most of the interpenetration problems, the shadowand sometimes still slightly interpenatrate the objects if moved too fast tho. Regarding the tangential friction, we have tried with both condim = 3 and condim = 6 but it doesn't seem to affect the simulation.

https://github.com/google-deepmind/mujoco/assets/127102416/03955233-faa6-4b12-8d93-d8ff1827ec57

The setting for the cylinder are the following: (these are the same for both the cube underneath the cylinder and the shadow hand)

We are not using any soft costraints for the hand movement but instead is a direct control with the position of the HTC VIVE Pro Eye controller.

[Balint-H]

So is the forearm's global position set simply using a mocap body? Then are there no joints in the hand? (As mocap bodies can't have child bodies)

Or do you set qpos directly manually?

If that is the case, the approach using a weld constraint detailed here is more appropriate for this purpose. The approach with a weld constraint constrains the maximum forces applied on the hand to make it track the mocap, otherwise massive contact forces can arise during interaction, which could prevent sliding/rolling.

[Bargez908]

What we did was firstly import the shadow hand xml file using MuJoCo import, than we did "create empty" -> "Add component" -> "Mj Mocap Body", then we also added as component to the same game object "Tracked Pose Driver" (as shown in the image below)

After creating the Mocap we put underneath it in hierarchy the shadow hand (named right_hand870 in the image below) so that it's attached to the Vive controller. ART and ARR are respectively the translational and rotational joints of the base of the hand (the foreharm) while geom 1 to 4 are the Mj Geom that compose it. the hierarchy is Tracker (mocap) -> right hand (the beginning of the body) -> forearm -> wrist -> palm -> fingers.

(this is the right hand component, a simple change of coordinates)

[Balint-H]

Okay in this case I can recommend trying the weld approach, that will give you more reliable interactions with the environment.

• Unparent the right hand from the mocap body
• Add a freejoint to the hand base
• Create a separate gameobject and add a MjWeld to it
• Set the tracker as Body1 and the hand base as Body2
• For the weld offset, assign any one transform that has 0 local position and no local rotation.

[Bargez908]

It is working now thank you so much! We can now correctly apply tangential forces and grasp objects.

There is one last problem tho, sometimes the simulation resets with the following error:

(we always had this issue but it was secondary to the tangential forces)

https://github.com/google-deepmind/mujoco/assets/127102416/365b90ad-2252-4aaf-8658-64a6abad137b

[Balint-H]

I'm glad it worked! Here's a discussion on simulation stability:

https://mujoco.readthedocs.io/en/stable/overview.html#divergence

If you haven't already, I recommend reducing the fixed timestep of your unity scene in your project settings. The default physics timestep in Unity is much larger than usually expected by mujoco models.

If a high timestep is a necessity, you can explore switching to a more stable integrator (e.g. implicit) in your mujoco options component. Lastly you can add a tiny amount to the armature field to all of your joints.

[Bargez908]

Alright, I will try those things, thank you again for your help!