Mujoco interface for SPEAR

[a2a4]

I've read in the newsletter that SPEAR has mujoco interface which can potentially support upto 100 fps. However, I could not find the relevant documentation to create this pipeline in the repo, apart from the script generate_mujoco_scene.py.

I've the following questions.

1) Does the current uploaded version of SPEAR provide the Mujoco interface? 2) Can we import MuJoCo sensors (camera, lidar), models, environments in SPEAR? 3) How to enable the interface/pipeline to mujoco? 4) Will this interface also support ROS2 pipeline for the robot control/navigation?

I humbly request your comments on the questions raised.

Thanks and Regards, Ajay.

[mikeroberts3000]

Hi. Great questions. In what newsletter did you read about our efforts to provide MuJoCo interoperability?

• Anyway, it is true that we're working on MuJoCo interoperability. There is already a bunch of working code in the main branch for this purpose, but it isn't well-documented yet. In the next couple of weeks, we expect to ship a new example app demonstrating some of this interoperability in our v0.5.0 release. See the movie below for a preview of what is currently possible using the code in main.
• The way MuJoCo interoperability currently works is as follows. You export one of our scenes from Unreal to MuJoCo, then you drive a physics simulation using MuJoCo, then at each time step of the simulation you communicate updated object poses to our Unreal executable using the spear Python module, and then use our Unreal executable to perform photorealistic rendering using the spear Python module. Most of these pieces are already working, but we're still finalizing some details.
• Our export code is in the spear/pipeline and the spear/cpp/unreal_projects/SpearSim/Python folders.
• So far, we have only attempted to export scene data from Unreal to MuJoCo. We haven't worked on importing model data from MuJoCo to Unreal, but this would certainly be useful. We intend to work on this in our v0.6.0 release. We have made several major architectural improvements that will make it much easier to import MuJoCo model data, but we haven't started work on actually importing complex MuJoCo models yet.
• ROS integration would certainly be useful, but we don't believe it is a fundamental abstraction. If you can {send, receive} {actions, observations} {to, from} {Unreal, MuJoCo} using appropriate Python interfaces, then it becomes easy for a user to build their own ROS wrapper (or LCM wrapper, or any other similar message passing wrapper) on top of those Python interfaces. But, on the other hand, if you can't {send, receive} {actions, observations} {to, from} {Unreal, MuJoCo}, then it becomes impossible to build any such message passing wrapper. Therefore, we think the Python interfaces for communicating with Unreal and MuJoCo are the fundamental abstractions, and this is where we are currently focusing our efforts. That being said, we are interested in developing example applications that integrate ROS. If you built a simple ROS integration demo, we'd be happy to include it as one of our example applications.

https://github.com/isl-org/spear/assets/2341965/80bf1138-0102-4905-b777-1f4f60bba14c

[a2a4]

Thank you so much Mike for taking your time to answer these questions. I read it in the circuit.