GSoC: Create Python bindings for MoveIt 2

[peterdavidfagan]

Hi there,

I am currently working on creating Python bindings for MoveIt 2 as part of my GSoC project.

The goals of my project are to:

1. Redesign and implement a Python API for MoveIt 2
2. Create a tutorial demonstrating the usage of this API for learning-based approaches
3. Implement feature requests that come from the MoveIt community

I will be adding further details to this issue page and my project page over time. Feel free to comment below or link feature requests that you feel may be relevant to this project.

Have a great rest of the day.

[AndrejOrsula]

Hello @peterdavidfagan,

Great to have someone working on this!

I also wanted to interface MoveIt 2 via Python for RL in the past, so I made pymoveit2 (previously under ign_moveit2_examples). It does not provide proper Python bindings, but rather a basic Python interface via ROS 2 middleware interface. Maybe there is something of relevance, so I decided to share it here.

---

Feel free to comment below or link feature requests that you feel may be relevant to this project.

If it is within the scope, here are some features that I would appreciate:

• A wrapper around MoveIt Servo (example).
• A simplified interface for grippers (support grippers with GripperCommand and JointTrajectoryController interfaces).
• The ability to reset the controllers after resetting/randomising robot joint state in paused/playing simulation. I found some approaches, but they are quite "hacky".
• Fully typed function arguments and return values for linting and auto-completion.
• Support for "overloaded" function arguments of different data types. E.g. the ability to pass either Iterable (List/numpy.ndarray/...) or geometry_msgs.msg.Point while setting the position goal. I am not sure how Type Adaption added to humble for rclcpp could fit into this context with rclpy.

[peterdavidfagan]

Hello @peterdavidfagan,

Great to have someone working on this!

I also wanted to interface MoveIt 2 via Python for RL in the past, so I made pymoveit2 (previously under ign_moveit2_examples). It does not provide proper Python bindings, but rather a basic Python interface via ROS 2 middleware interface. Maybe there is something of relevance, so I decided to share it here.

Feel free to comment below or link feature requests that you feel may be relevant to this project.

If it is within the scope, here are some features that I would appreciate:

• A wrapper around MoveIt Servo (example).
• A simplified interface for grippers (support grippers with GripperCommand and JointTrajectoryController interfaces).
• The ability to reset the controllers after resetting/randomising robot joint state in paused/playing simulation. I found some approaches, but they are quite "hacky".
• Fully typed function arguments and return values for linting and auto-completion.
• Support for "overloaded" function arguments of different data types. E.g. the ability to pass either Iterable (List/numpy.ndarray/...) or Pose while setting the pose goal. I am not sure how Type Adaption added to humble for rclcpp could fit into this context with rclpy.

Thanks @AndrejOrsula,

The Python library you already created looks great, it is certainly relevant as developing user friendly interfaces is an important part of this project.

I appreciate the feature requests, I will take these into account and work towards including these features. Looking forward to discussing further, I'll share design details here and on Discord while progressing with development. It would be great to receive your feedback.

[tlpss]

Hi @peterdavidfagan

Very glad to see someone is going to work on this!

I have also created a "python bridge" for some features of Moveit2 that I wanted to use to control our UR robots, but the lack of an easy-to-use and well documented python interface to Moveit2 has been the main bottleneck for our research group to start using Moveit2 in our research, as it creates a barrier for researchers that are not experienced with ROS.

Things that we would be looking for include:

• standard aspects such as access to the planning interface and scene interface (the Move Group Interface basically).

• the ability to manually define waypoints and execute them through Moveit for IK and collision checks. (We often want to specify the end-effector trajectory exactly as our research group works quite a lot on deformable object manipulation, for which not only the pick and place pose are important). We have been using the Cartesian Interpolator in the Move Group Interface to achieve this but I have seen some discussions on removing this as it easily gets stuck. We would however still really like to be able to specify exact trajectories for the EEF to follow.

• The wrapper around Moveit Servo, as suggested by @AndrejOrsula. And more generally, the ability to specify goal poses, waypoints,... in a more pythonic format (numpy arrar/list) instead of having to create a Twist message. (Numpy arrays are typically what you want to use for Data-driven action command interfaces).

• Best practice examples on how to use the python interface of Moveit2 for "learning-based" approaches. Basically: how to start when you want to use Moveit2 for training a robot to perform a particular task. Lots of simulators are starting to include ROS interfaces (Unity, ISAAC,...) which has the potential to use the same ROS stack in a simulator and for real robots, but there are no well-documented examples on how to get started with this afaik. For researchers without ROS experience, this would be essential. Some (simulator-independent) Moveit2-Gym interface template would be of great value. An example of how to use moveit servo if you want fine-grained EEF control for the policy as wel as an example of using Moveit for interactively learning grasp poses (and then executing them with Moveit) would be very interesting.

[peterdavidfagan]

Hi @tlpss,

Thank you for these points. I have taken note of these and will look to incorporate the outlined functionalities into the Python API that gets created. It would be great to have you test the initial library that results from this project.

Currently we are deciding what to bind and how the API should be structured. At the moment it looks like the functionality of the moveit_cpp library will be binded. The overall API design is still not finalized. I hope to provide demos using Isaac Sim so maybe this will be helpful for your third point.

[tlpss]

It would be great to have you test the initial library that results from this project.

Feel free to ping me, I would be interested to take a look.

Best of luck with the project!

[peterdavidfagan]

Hi everyone,

The development of a library that is called moveit_py has begun; initially this library will consist of binding functionalities of moveit_cpp and creating useful python interfaces using these functionalities. As additional functionalities are added to moveit_cpp these will also be considered for moveit_py. For those who are interested in tracking the development work for this library the moveit_py branch on my fork should remain up to date.

I also have another repository moveit_py_example which will demonstrate examples of using the library. There's only one example there for the moment but I will be adding many more in the coming weeks (including examples which leverage isaac-sim).

Finally I have started a project to track development work which can be found here. I'll be adding tasks to this soon.

Other than this update, I may share some of the interfaces in this thread to get your feedback on these so expect further updates in the coming weeks. Have a great rest of the day.

[marc-wittwer]

Awesome. @peterdavidfagan I will follow your progress :raised_hands:

The python interface will lower the entry barrier for our research group as well.

We will mainly use:

• Setting pose goals
• Setting collision objects into the planning scene
• Setting orientation constraints of the end effector while following a trajectory (e.g. moving a cup with liquid inside)

Best of luck with the project also from my side!

[peterdavidfagan]

Thanks @marc-wittwer, I am really looking forward to the first release of the library. I'll be sure to take into account the use cases you mentioned above. It is great to have so much interest in the library already thank you for sharing the above.

[larrylu0426]

Hi everyone, I'm working on a project about Panda's arm, and it depends on Python3 and ROS2/Moveit2/Ubuntu20.04. So I refactor the planning interface module with pybind11 on the foxy branch. Here's my repo link: https://github.com/larrylu0426/moveit2. I keep all C++ interfaces and adapt the original Python interfaces. I have applied it in my project. So you can use it and give me many reviews :).

MoveIt Command is ready to refactor.

[peterdavidfagan]

Hi @larrylu0426,

The above looks great, it sounds like you are interested in the same topics as this GSoC project.

You can view the current status of the development here and a basic example of its usage here. I am starting to work on bindings for the planning_scene_interface as part of the package linked above. The bindings you created with pybind11 are similar to the ones I am developing, except the bindings I am creating lie in a separate package and there are other minor difference (e.g. how ros messages are being handled).

If you are up for it we could hop on a call and discuss these bindings? I believe it may make sense to incorporate some of your code into the package I am creating as part of my GSoC project.

[larrylu0426]

Hi everyone, I have done the refactoring of the whole planning interface module and most parts of the moveit_commander module on main branch. Also, I have done a basic test by myself. Pls feel free to try it in your project :). The rest of remained code in moveit_commander will not be refactored from my side. I'm expecting the Peter's work. I think it will provide more powerful things for Python developers. :)

[peterdavidfagan]

Hi @larrylu0426,

The above refactoring is great and I think it will be useful for users of the planning_interface and moveit_commander.

It would also be great to get your continued involvement in moveit_py, I'd be happy to schedule a call to discuss further. Also if you don't mind I would like to ask if you would be interested in testing the initial release I make of moveit_py?

[larrylu0426]

Hi @larrylu0426,

The above refactoring is great and I think it will be useful for users of the planning_interface and moveit_commander.

It would also be great to get your continued involvement in moveit_py, I'd be happy to schedule a call to discuss further. Also if you don't mind I would like to ask if you would be interested in testing the initial release I make of moveit_py?

Great! I will try to use your released version for my work and report the usage result. We can keep in touch by email due to the time difference. 😄

[peterdavidfagan]

Hi everyone,

Thanks for being patient on the first release of moveit_py. I wished to provide a brief update on the current status and some exciting functionalities that become possible with the use of Python.

• The main functionalities of the C++ library moveit_cpp are now binded
• Many of the functionalities of RobotState and RobotModel are now binded
• Interacting with the planning scene via the planning scene monitor is now supported
• Support for retrieving function signatures and auto-completion within IDE (we have added stub files to the project, they still need to be completed)
• The option to use Jupyter Notebooks to interactively write and execute code is being tested

I hope to push a broader set of changes this weekend related to the above points and start writing tutorials for the currently available functionality. I will send a notification here when all these changes have been pushed. I'll also try to provide more frequent updates from now until the official release of the library (which will likely be another while since documentation and tests need to be written while the library also needs to mature sufficiently and achieve a standard that can be approved and merged by my mentors).

In the future, I also hope to add support for:

• APIs for interacting with a Servo Node
• Examples/support for reinforcement learning applications

Hopefully this brief update gives you an idea of the current status.

Have a great rest of the day and hopefully more updates to come soon.

[lorepieri8]

@peterdavidfagan When do you think the bindings will be released? Happy to test them and share feedback!

[peterdavidfagan]

Hi @lorepieri8,

The GSoC project runs until the end of October so at the latest I would expect to release the Python library by then.

I am currently working on docs and unit tests with the goal of having a library that can be released by the end of this month. This depends on a review from the current maintainers of the MoveIt project. If you want to start using the library you can build the library from the moveit_py branch on my fork of the moveit2 repository. The library is currently compatible with the rolling ROS distro.

Happy to test them and share feedback!

Thanks this is much appreciated, if you start using the library in its current state and have feedback it would be great to hear your feedback as it will help inform further development of the library.

[peterdavidfagan]

Hi @AndyZe,

I am starting to work on a python client for servo once again to allow users to leverage the Servo module you created via Python (mainly for learning-based approaches). Some use cases require sending commands for both the position change of the end effector and basic gripper commands. For instance in the below paper there is an example of multiple command types that are specified at each timestep:

source: https://sites.google.com/view/bc-z/home

One item I wished to ask you about was support for various message types. At the moment it seems like control messages such as TwistStamped and JointJog are supported.

I was wondering if additional message types can or are already supported such as GripperCommand?

A secondary question is dealing with the timing of execution of commands. Ideally, it would be nice to execute commands for the eef position and gripper simultaneously. Is this currently supported? Do I have to setup two Servo nodes to achieve this functionality?

Thanks in advance for any guidance you can provide on the above.

[AndyZe]

Nice. No, GripperCommand isn't used.

There are a few services to allow/disallow motion in certain dimensions, but I don't think it's important enough to port to Python:

test/servo_launch_test_common.hpp: client_change_drift_dims_ = node_->create_client<moveit_msgs::srv::ChangeDriftDimensions>(

src/servo_calcs.cpp: control_dimensions_server_ = node_->create_service<moveit_msgs::srv::ChangeControlDimensions>(

Nothing else comes to mind

[peterdavidfagan]

I would be happy to work towards adding support for the above functionality with your guidance. It would be helpful for enabling learning-based approaches. I also feel this is something other users would find valuable.

Would it be feasible to setup sometime to discuss what is required to support such functionality with Servo?

Update: thanks @AndyZe.

[ronniethehood]

Hi @peterdavidfagan, this looks great. Any update on when it might be released?

[peterdavidfagan]

Hi @peterdavidfagan, this looks great. Any update on when it might be released?

Thanks @ronniethehood, apologies for the delay in the release of the library. The initial plan was to release at the end of this month, this is now more likely to be in mid-November. For context I am working on unit testing, tutorials and cleaning up the code. There will also be reviews from some of the maintainer team which should lead to additional improvements to the python library code.

Once the initial release is made there will be continued support for feature requests and development so if you start using the library after we release it and have a feature that you wish to be supported I am happy to work on this.

[vuva]

Hi @peterdavidfagan, this looks great. Any update on when it might be released?

Thanks @ronniethehood, apologies for the delay in the release of the library. The initial plan was to release at the end of this month, this is now more likely to be in mid-November. For context I am working on unit testing, tutorials and cleaning up the code. There will also be reviews from some of the maintainer team which should lead to additional improvements to the python library code.

Once the initial release is made there will be continued support for feature requests and development so if you start using the library after we release it and have a feature that you wish to be supported I am happy to work on this.

Hi @peterdavidfagan , is time parameterization supported in the initial version of the python binding? And sorry to ask this again, when will you release it? Thank you

[peterdavidfagan]

Hi @vuva,

Apologies that you have had to ask this. Time parameterization will eventually be supported, it is not in the current set of code for the library but will be added.

The initial plan was to release the library by the end of GSoC, unfortunately, this deadline was not met. I have set aside time in December and January just for the completion of this library so I hope to have code ready to be merged by the new year.

[metanav]

Any ETA for the release?

[peterdavidfagan]

Hi @metanav,

The current pr has been rebased to include the latest set of commits on the main branch of the MoveIt repository so the library code should be usable with the latest changes.

I am finishing writing the tutorials and ensuring the API documentation is accurate this week. I may also make some API changes this week but otherwise, I would hope that the code can be reviewed and merged soon. I expect changes to be complete on my side and ready for review by the end of this week but I am unsure how long it will take to be reviewed and merged thereafter.

Also feature requests are welcome once the code has been merged.

[sjahr]

@peterdavidfagan I guess this is done :tada: Can we close it?

[peterdavidfagan]

Hi @sjahr,

Apologies for delayed responses on my side recently. There are more features and improvements I wish to add to the library but I think this task can be closed for now. I will follow up with separate items for anything else I work on.