search

nasa-jpl / rosa

ROSA 🤖 is an AI Agent designed to interact with ROS1- and ROS2-based robotics systems using natural language queries. ROSA helps robot developers inspect, diagnose, understand, and operate robots.
https://github.com/nasa-jpl/rosa/wiki
Apache License 2.0
673 stars 49 forks source link

Code for controlling Spot in simulation #27

Open AayushManShrestha opened 1 week ago

AayushManShrestha commented 1 week ago

In the YouTube demo video, a Spot robot was being controlled via ROSA. I couldn't find the code used in the demo video. It would be helpful if code used to control a Spot robot in simulation was available. Thank you.

RobRoyce commented 1 week ago

Setting up ROSA for physical robot control is implementation-specific. That's one of the reasons we decided to be un-opinionated when it comes to uplink command and control.

For the demo, we basically gave ROSA an Xbox controller. It publishes joystick messages (axes and button presses) to the joy topic, which gets interpreted by the teleop_twist_joy node, which then publishes twists to cmd_vel. We also have a node that multiplexes any /joy publishers to ensure ROSA commands can always be overridden by human input (JoyMux will override ROSA commands when it detects ESTOP button on Xbox controller).

  graph TD;
      ROSA --> |/joy| JoyMux{JoyMux};
      XboxCtrl --> |/joy| JoyMux{JoyMux};
      JoyMux --> |/joy_cmd| TeleopJoy;
      TeleopJoy --> |/cmd_vel| Spot

Can you tell me how you're currently controlling the robot (e.g. publishing to /cmd_vel)? And which capabilities you're interested in from the demo? We will consider creating a default Spot embodiment if it can be generalized to cover most implementations.

Alternatively, you can check out the publish_twist_to_cmd_vel tool implemented for the turtle_agent, which can easily be adapted as a custom tool for your use case.

AayushManShrestha commented 1 week ago

I am currently controlling the Spot robot by publishing velocity commands to the /spot/cmd_vel topic, which are then processed by a ROS node to execute the desired movements using the Spot SDK. If you are considering to create a general implementation, then using Spot with arm would be better.

Image

I am interested in the following capabilities:

  1. Controlling Spot's movement using natural language instructions
  2. Controlling Spot arm to manipulate object
  3. Capturing visual data from the simulation environment
  4. I don't know if this goes beyond the scope of the project but integrating VLMs would be nice