Pass a C++ mjData to python with pybind11, MuJoCo as a C++ and python library

[pkrack]

Hi,

I am a student trying to use MuJoCo for collision detection and trajectory planning for a real robot, and for reinforcement learning experiments on the same simulated robot.

I need the flexibility to implement performance critical code in C/C++ (e.g. IK & trajectory planning), while being able to easily experiment with python (e.g. for reinforcement learning).

I compile both the C dynamic library and the bindings with scikit-build-core and link my own code against the same dynamic library than the MuJoCo python bindings. I write python bindings for my code using pybind11 as well.

I would like to interact with the simulation from python and C++. In this context, I am trying to return, from C++ to python, a pointer to a mjData struct. Directly returning a reference to the shared pointer does not work.

py::class_<SimState>(pymodule, "SimState")
.def_readonly("collision", &SimState::collision)
...
.def_readonly("physics_state", &SimState::mjdata); // <- What should I return here?

Python complains that it does not know about the mjData_

I have tried including structs.h returning a MjDataWrapper, but I get an unknown symbol error, even after adding PYBIND11_EXPORT to the wrapper, the base wrapper class and the raw mjData object.

In this issue, @saran-t seems to suggest that the MjDataWrapper is not necessary and I should simply return opaque pointers, which I tried with PYBIND11_MAKE_OPAQUE, but python still complains about an unknown type mjData_

I am now out of ideas and grateful for any help.

[saran-t]

The easiest thing in your case is to instantiate the MjData object in Python and grab the underlying pointer as a void, pass it as a long int to pybind11 and cast it back to mjData in C++. That way you never need to deal with MuJoCo types across the Python/C++ boundary.

Alternatively, if you are interested in doing some fairly advanced pybind11 work, you can help us figure out how to register custom casters so that pybind11 understands that mjData should be casted via MjDataWrapper.

[pkrack]

Thank you for the quick answer and the great library. The fact that the pybind11 docs about custom casters mention the "intricacies of python reference counting" is scaring me off a bit. If I get to it will prepare a PR / an issue (have not read the contributing guidelines yet).

[peterdavidfagan]

@pkrack did you successfully link the python binding MjWrapper classes to your pybind library? It seems like yes based on:

I compile both the C dynamic library and the bindings with scikit-build-core and link my own code against the same dynamic library than the MuJoCo python bindings.

Would it be possible to share this code if you have it (I've been finding it relatively involved to link against the structs wrappers, I am in the process of trying to create a static library from the binding code that I can link in my own code. It would be really helpful to reference what you did as this will solve the problem I am currently dealing with). My strategy was to return the MjModelWrapper from my binding method for getting a readonly property similar to the following method:

mujoco::python::_impl::MjModelWrapper FrankaMJROS::get_model()
{
  return *mujoco::python::_impl::MjModelWrapper::FromRawPointer(this->m);
}

I think this should work right as it returns the same type for which there exists a binding in the official python library. Maybe it requires manually importing the official bindings in your binding library but otherwise I think this return should be defined and work.

[peterdavidfagan]

The easiest thing in your case is to instantiate the MjData object in Python and grab the underlying pointer as a void, pass it as a long int to pybind11 and cast it back to mjData in C++. That way you never need to deal with MuJoCo types across the Python/C++ boundary.

This approach might actually makes more sense in my use case, I am going to try it out.