How to use Mujoco in real time system, especially when hardware-in-the-loop simulation is involved

[areofire]

Hi,

I'm a engineer)and I'm trying to use MuJoCo for real time simulation.

I'm looking for some help with how to how to use Mujoco in real time system, especially when hardware-in-the-loop simulation is involved.

[nimrod-gileadi]

Do you have any specific questions?

If by "real time" you just mean fast enough to execute on a real robot, then MuJoCo should be good. If you mean simulating at wall clock rate, you could look at the source code for the simulate library which does this.

However, if you mean "real time" in terms of guaranteed deadlines for computation, MuJoCo doesn't offer such guarantees, and I don't know of an example of a strong real time system using it. Simulation time varies depending on the number of active contacts at a given moment, and collision detection time varies based on the positions of bodies.

However, physics stepping doesn't result in any heap allocations, and you can set various limits to control the maximum computation time per step, e.g:

• Only evaluating collisions and contacts between predefined pairs of geoms using collision="predefined" in the option tag, or setting a maximum number of contacts with nconmax.
• Adjusting the maximum number of solver and collision detection iterations with iterations and mpr_iterations.

[areofire]

Thank you for your reply.

I want to build a hardware-in-the-loop simulation which use physical hardware such as Nvidia's Jetson TX2 as a real-time controller and another pc as a dynamics host. In the dynamics host run a mujoco model and the model's drive torque comes from the real-time controller.

Use UDP or CAN for data transmission between the controller and pc. The controller calculates the control torque in real time and send it to dynamics host. The model in the dynamics host moves according to the driving torque and feeds back the state confidence.

So, I want to konw whether mujoco can be used as a real-time model and run on a real-time system such as vxWorks or other real-time systems.

[nimrod-gileadi]

So it sounds like you want both:

1. Simulation to be run at 1 simulation second per real world second. Let's call this "real-world time" rather than "real time".
2. Build your dynamics host using a real-time OS.

On the first requirement, on most robotics setups which don't include rendering camera sensors will comfortably be simulated much faster than real time.

On the second one, we don't have the expertise on the team to advise you. Maybe someone else in the community knows better, or if there is a list of requirements for a library to be run in vxWorks, we could look at it.

From hearing your description, I would say that maybe running the simulation itself in a real-time context is not so important: what's important is that the dynamics host responds to requests in real time, to satisfy the controller. So you could have a background simulation thread running close to real-world time and updating the physics state as it goes, and the server could take whatever the latest computed state is when the request comes in, without blocking the request.

[JeanElsner]

From hearing your description, I would say that maybe running the simulation itself in a real-time context is not so important: what's important is that the dynamics host responds to requests in real time, to satisfy the controller. So you could have a background simulation thread running close to real-world time and updating the physics state as it goes, and the server could take whatever the latest computed state is when the request comes in, without blocking the request.

We did exactly this, running MuJoCo with hardware in the loop at 1KHz and even interacting haptically with the simulation, it worked quite well: https://www.youtube.com/watch?v=VkGps9ZC2jY

Alternatively you could use dm_robotic's MoMa which offers an abstraction layer to interface with your hardware. However, the latter is not suitable to compute low-level actuation in simulation but rather to produce kinematic control signals (e.g. join velocities).

[kevinzakka]

@JeanElsner That's an awesome video!

[areofire]

what's important is that the dynamics host responds to requests in real time, to satisfy the controller.

Yes, that's the questions i want to ask. I watch the video and read the paper, it's a outstanding working and I will try it. Many thanks, @Collaborator, @JeanElsner.