Real world control stuck on "Waiting for '/controller_manager' services to be available"

[haochengao]

Hey, big thanks for creating this great project. I'm having an issue getting the real world control up and running. Here's the command I'm trying: ros2 launch ar_hardware_interface ar_hardware.launch.py ar_model:=ar4_mk3 calibrate:=False include_gripper:=False which results in

[INFO] [launch]: All log files can be found below /root/.ros/log/2024-07-27-13-08-43-909553-orangepi5plus-36114
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [ros2_control_node-1]: process started with pid [36119]
[INFO] [spawner-2]: process started with pid [36121]
[INFO] [robot_state_publisher-3]: process started with pid [36123]
[INFO] [spawner-4]: process started with pid [36125]
[ros2_control_node-1] [WARN] [1722056924.564253140] [controller_manager]: [Deprecated] Passing the robot description parameter directly to the control_manager node is deprecated. Use '~/robot_description' topic from 'robot_state_publisher' instead.
[ros2_control_node-1] [INFO] [1722056924.564765300] [resource_manager]: Loading hardware 'ar4_mk3'
[ros2_control_node-1] [INFO] [1722056924.567247347] [resource_manager]: Initialize hardware 'ar4_mk3'
[ros2_control_node-1] [INFO] [1722056924.567312971] [ar_hardware_interface]: Initializing hardware interface...
[ros2_control_node-1] [INFO] [1722056924.569056821] [teensy_driver]: Successfully connected to serial port /dev/ttyACM0
[ros2_control_node-1] [INFO] [1722056924.569124487] [teensy_driver]: Waiting for response from Teensy on port /dev/ttyACM0
[robot_state_publisher-3] [INFO] [1722056924.572851933] [robot_state_publisher]: got segment base_link
[robot_state_publisher-3] [INFO] [1722056924.573076222] [robot_state_publisher]: got segment ee_link
[robot_state_publisher-3] [INFO] [1722056924.573092263] [robot_state_publisher]: got segment link_1
[robot_state_publisher-3] [INFO] [1722056924.573101888] [robot_state_publisher]: got segment link_2
[robot_state_publisher-3] [INFO] [1722056924.573114430] [robot_state_publisher]: got segment link_3
[robot_state_publisher-3] [INFO] [1722056924.573125221] [robot_state_publisher]: got segment link_4
[robot_state_publisher-3] [INFO] [1722056924.573136013] [robot_state_publisher]: got segment link_5
[robot_state_publisher-3] [INFO] [1722056924.573149137] [robot_state_publisher]: got segment link_6
[robot_state_publisher-3] [INFO] [1722056924.573162554] [robot_state_publisher]: got segment world
[spawner-2] [INFO] [1722056926.922269959] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' services to be available
[spawner-4] [INFO] [1722056926.922686453] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available

The last 2 messages repeat indefinitely. When I look at ros2 node list here is what I see:

/controller_manager
/robot_state_publisher
/spawner_joint_state_broadcaster
/spawner_joint_trajectory_controller

and ros2 service list:

/controller_manager/describe_parameters
/controller_manager/get_parameter_types
/controller_manager/get_parameters
/controller_manager/get_type_description
/controller_manager/list_parameters
/controller_manager/set_parameters
/controller_manager/set_parameters_atomically
/robot_state_publisher/describe_parameters
/robot_state_publisher/get_parameter_types
/robot_state_publisher/get_parameters
/robot_state_publisher/get_type_description
/robot_state_publisher/list_parameters
/robot_state_publisher/set_parameters
/robot_state_publisher/set_parameters_atomically
/spawner_joint_state_broadcaster/describe_parameters
/spawner_joint_state_broadcaster/get_parameter_types
/spawner_joint_state_broadcaster/get_parameters
/spawner_joint_state_broadcaster/get_type_description
/spawner_joint_state_broadcaster/list_parameters
/spawner_joint_state_broadcaster/set_parameters
/spawner_joint_state_broadcaster/set_parameters_atomically
/spawner_joint_trajectory_controller/describe_parameters
/spawner_joint_trajectory_controller/get_parameter_types
/spawner_joint_trajectory_controller/get_parameters
/spawner_joint_trajectory_controller/get_type_description
/spawner_joint_trajectory_controller/list_parameters
/spawner_joint_trajectory_controller/set_parameters
/spawner_joint_trajectory_controller/set_parameters_atomically

I'm not sure why this is happening since it looks like the controller_manager services are up and running?

[se1exin]

The log message below about waiting for /dev/ttyACM0 looks suspicous to me.

[ros2_control_node-1] [INFO] [1722056924.569124487] [teensy_driver]: Waiting for response from Teensy on port /dev/ttyACM0

If you enable calibration with calibrate:=True, does the arm move and run it's calibration sequence?

[ycheng517]

Se1exin is correct, looks like Teensy isn't responding. You can troubleshoot this using Arduino's Serial Monitor, see if you can connect to the teensy (with baud rate 9600) and send commands to it. i.e. sending JC should start the calibration routine, and sending JP should return the current position.

[zhelfer0306]

I'm also having the exact same issue. So far, it seems like something in the sketch is mishandling serial communication. I created a very simple test program over the same serial connection that echoes back what I sent via the serial monitor that worked, so that at least isolates the issue to the sketch (I'm using AR4_MK2_MK3.ino).

[ycheng517]

It looks like others have no issues with serial communication with the MK3, see: https://github.com/ycheng517/ar4_ros_driver/issues/15 . I don't think it's a systematic issue as I know quite a few people that have gotten everything to work with MK2 and the original version, and MK3 shouldn't be that different.

I again recommend trying to use the Arduino Serial Monitor to communicate with the Teensy 4.1 running the firmware sketch. I also suggest getting the robot working with the official Annin Robotics Control Software and ensure there's no issues there.

[se1exin]

One thing I find that often messes with microcontrollers is the USB cable, both it's length and the quality of the cable itself. I have a handful of USB cables that work with all my devices except arduinos/ESPs/teensies, and it seems it's mostly the thicker, short length, cables that do work.

I also get intermittent disconnections when using a USB hub, so instead I try to plug directly into my machine when possible. USB extension cables nearly always cause issues too - I recently upgrade to an expensive 2 meter long one and it does work though.

Finally, noise in the power circuits of your circuits (e.g. in the robot arm circuitry) can cause flakely behavior for USB comms and other functions of the microcontrollers. On my ARM-MK2 I have put capacitors on every single wire that goes to a motor, and I've also wired absolutely every separate circuit to a common ground.

In summary there a many hardware-level things that could be causing the USB to not work 100%, I would be checking my circuits and cables before looking at software as being the cause.

Edit: Here is a pic of my teensy with the capacitors, they're a bit hard to see so I've circled them in red:

[zhelfer0306]

Thanks for the replies. Serial communication is working correctly for me with the latest MK3 sketch pushed yesterday.

[ycheng517]

glad to hear it's working @zhelfer0306 , and nice tips @se1exin