search

google-deepmind / mujoco

Multi-Joint dynamics with Contact. A general purpose physics simulator.
https://mujoco.org
Apache License 2.0
8.09k stars 810 forks source link

Free jont pose control of Shadow Hand using MuJoCo viewer and API #1282

Closed vmstavens closed 10 months ago

vmstavens commented 10 months ago

Hi,

I'm an engineer at a University and I'm trying to use MuJoCo for tactile perception using the [Shadow E3M5](https://github.com/google-deepmind/mujoco_menagerie/blob/main/shadow_hand/README.md). To do so, I need to pick and place a box in my environment.

I'm looking for some help with finding a way to move the Shadow Hand around in my simulated world using both the Python Bindings and the MuJoCo Simulation Viewer. The behavior I am looking for is the same as if the hand was mounted to a manipulator but without the manipulator itself.

As an example I should have something like a slider for each DoF under the Control panel with names such as pos_x, pos_y, pos_z, roll, pitch, yaw, that dictate the current pose of the hand's, let's say forearm frame. When a pose is set I also want the pose to be kept, rather than the hand falling due to gravity.

My current progress is

  1. The hand is floating rather than falling.

  2. I can control the orientation about the forearm's frame's x-axis by

    1. Defining a actuator class named rot_x:

      <default>
  <default class="right_hand">
    <mesh scale="0.001 0.001 0.001"/>
    <joint axis="1 0 0" damping="0.05" armature="0.0002" frictionloss="0.01"/>
    <position forcerange="-1 1"/>

    <! -- my addition -->
    <default class="rot_x">
        <joint axis="1 0 0" limited="false"/>
        <position kp="100" kv="10" ctrlrange="-3.14 3.14" forcerange="-10 10"/>
    </default>

    <default class="wrist">
      <joint damping="0.5"/>
      <default class="wrist_y">
      ...

    2. Defining a free joint free_joint_x in the robot's rh_forarm body:

      <worldbody>
  <body name="rh_forearm" childclass="right_hand" quat="1 -1 1 -1" pos="1 1 1">
      <! -- my addition -->
      <joint name="free_joint_x" stiffness="0" damping="0" frictionloss="0" armature="0"/>
      ...

    3. Adding the controller to the actuators:

      <actuator>
<! -- my addition -->
  <position name="ros_x" joint="free_joint_x" class="rot_x"/>

  <position name="rh_A_WRJ2" joint="rh_WRJ2" class="wrist_y"/>
  <position name="rh_A_WRJ1" joint="rh_WRJ1" class="wrist_x"/>
    ...

But as mentioned this is not enough to solve my problem. I have tried to define new actuators and joints for each DoF but MuJoCo throws the warning: WARNING: Inertia matrix is too close to singular at DOF 0. Check model. Time = 0.0000.

while my Shadow Hand curls up in nightmare fueling configurations.

I am running my simulation in a virtual environment on Ubuntu 20.04. The environment is running

All hint, help, or guidance is very much appreciated, and thanks in advance.

yuvaltassa commented 10 months ago

The standard and recommended way to do this is with a mocap body. Read that link carefully as there are a few caveats.

One of the caveats is that mocap body poses cannot be controlled with an actuator, so if you want sliders to control your arm position that won't work. If that's a requirement then instead of a free joint put a series of 3 sliders and 3 hinges (maybe 2 hinges are enough?) at the base and add position actuators with high kp and kv (remember the respective units for sliders are w.r.t to meters and for hinges w.r.t radians!). You could also use a free joint but then you'll need to define a vector gear attribute. However, that can only work for position, not orientation, due to the quaternion in the free joint. This is why a series of hinges is the best way to do orientation controlled generally.

Finally, the error you're seeing has nothing to do with any of this. Do you have a freejoint and some other joint in the same body? That can't work unless you throw in some armature, otherwise, as the error says, the inertia matrix becomes singular.

vmstavens commented 10 months ago

Hi Yuval,

Thank you so much for your quick and thorough response! I will try out your different suggestions and see where that takes me.

kevinzakka commented 10 months ago

Hi @vmstavens, for a concrete implementation of adding slider joints for the translational movements and hinge joints for the rotational movements of the hand, see the RoboPianist implementation here.