search

google-research / robopianist

[CoRL '23] Dexterous piano playing with deep reinforcement learning.
https://kzakka.com/robopianist/
Apache License 2.0
567 stars 46 forks source link

Integration with the Unity game engine #13

Open SherAlex1998 opened 1 year ago

SherAlex1998 commented 1 year ago

I'm going to reproduce the hands movement in Unity. Started with Twinkle Twinkle Little Star example.

At first, I saved the scene with all the resources:

export_with_assets(
      task.root_entity.mjcf_model,
      out_dir="/tmp/robopianist/piano_with_shadow_hands",
      out_file_name="scene.xml",
)

At second, I installed Unity plug-in from MuJoCo website. At third, I changed scene assets from obj format to stl format like user pkrack wrote here Finally, Unity has able to load this scene. But when I louched it I recived errors:

Element 'joint', line 1

NullReferenceException: Failed to create Mujoco runtime.
Mujoco.MjScene.StepScene () (at MuJoCo/Runtime/Components/MjScene.cs:345)
Mujoco.MjScene.FixedUpdate () (at MuJoCo/Runtime/Components/MjScene.cs:98)

The first occurrences of "range" in the xml debug file contain a comma in the values, not a period. Maybe that's the problem. But I do not know how to change the generation of this file)

<mujoco>
  <worldbody>
    <body pos="0 0 0" quat="-1 0 0 0" gravcomp="0" name="lh_shadow_hand/_0">
      <body pos="0.4 -0.15 0.13" quat="-0.5 -0.5 0.5 0.5" gravcomp="1" name="lh_shadow_hand/lh_forearm_1">
        <inertial pos="0 0 0.09" quat="-1 0 0 0" mass="3" diaginertia="0.0138 0.0138 0.00744" />
        <joint type="slide" pos="0 0 0" axis="-1 0 0" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="67.4762" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="-0.4605 0.7605" name="lh_shadow_hand/forearm_tx_3" />
        <joint type="slide" pos="0 0 0" axis="-1.192093E-07 0 1" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="67.4762" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="0 0.06" name="lh_shadow_hand/forearm_ty_4" />
        <geom density="1000" type="mesh" mesh="mesh_6" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="0" conaffinity="0" group="2" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_0_5" />
        <geom density="1000" type="mesh" mesh="mesh_8" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="0" conaffinity="0" group="2" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_1_7" />
        <geom density="1000" type="mesh" mesh="mesh_10" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="1" conaffinity="1" group="3" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_2_9" />
        <geom density="1000" type="box" size="0.035 0.035 0.035" pos="0 -0.01000001 0.181" quat="0.9249092 0 0.3801881 0" priority="0" contype="1" conaffinity="1" group="3" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_3_11" />
        <site type="box" size="0.001 0.001 0.001" pos="0 0 0" quat="-1 0 0 0" name="lh_shadow_hand/forearm_tx_site_12" />
        <site type="box" size="0.001 0.001 0.001" pos="0 0 0" quat="-1 0 0 0" name="lh_shadow_hand/forearm_ty_site_13" />
        <body pos="0 -0.01000001 0.21301" quat="-1 0 0 0" gravcomp="1" name="lh_shadow_hand/lh_wrist_14">
          <inertial pos="0 1.490116E-08 0.029" quat="0.5 0.5 0.5 0.5" mass="0.1" diaginertia="6.4E-05 4.38E-05 3.5E-05" />
          <joint type="hinge" pos="0 0 0" axis="-1.192093E-07 -1 0" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="0.5" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="-30,00002 10" name="lh_shadow_hand/lh_WRJ2_16" />

...

I attached this file to post, you may look it.

As a result, maybe it would be better to somehow copy the frame-by-frame hand positions to make an animated clip? Where can I get these values from an xml file generated by a neural network? Importing the environment as it is in Unity seems to be a more confusing task. debug.txt