search

RethinkRobotics / baxter_simulator

Gazebo Simulation interface for the Baxter Research Robot
http://sdk.rethinkrobotics.com/wiki/Baxter_Simulator
BSD 3-Clause "New" or "Revised" License
85 stars 95 forks source link

RSDK1.1 Baxter_gazebo Could not stop controller 'left_joint_velocity_controller' since it is not running #81

Closed Petrox closed 8 years ago

Petrox commented 8 years ago

We have a project that depends on baxter_sdk1.1, ubuntu 14.04, ros indigo. (the project is in codefreeze state, so no 1.2 upgrade possible this time, not even apt-get upgrade allowed)

While we do have a working baxter robot, but we're trying to simulate baxter under gazebo 2.2.6 (from osrf repo).

I even set the export LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH" trick in my .bashrc, as was suggested on the mailing list, and it solved some error messages about missing robot_state before...

when running roslaunch baxter_gazebo baxter_world.launch

[ INFO] [1455549597.698834625, 1.974000000]: Initializing BaxterEffortController with 7 joints.
[ INFO] [1455549597.699059826, 1.974000000]: Loading sub-controller 'right_e0_controller', Namespace: /robot/right_joint_effort_controller/joints/right_e0_controller
[ INFO] [1455549597.704393518, 1.978000000]: Loading sub-controller 'right_e1_controller', Namespace: /robot/right_joint_effort_controller/joints/right_e1_controller
[ INFO] [1455549597.717833827, 1.984000000]: Loading sub-controller 'right_s0_controller', Namespace: /robot/right_joint_effort_controller/joints/right_s0_controller
[ INFO] [1455549597.726747593, 1.988000000]: Loading sub-controller 'right_s1_controller', Namespace: /robot/right_joint_effort_controller/joints/right_s1_controller
[ INFO] [1455549597.733626383, 1.994000000]: Loading sub-controller 'right_w0_controller', Namespace: /robot/right_joint_effort_controller/joints/right_w0_controller
[ INFO] [1455549597.744691360, 1.997000000]: Loading sub-controller 'right_w1_controller', Namespace: /robot/right_joint_effort_controller/joints/right_w1_controller
[ INFO] [1455549597.761715324, 2.014000000]: Loading sub-controller 'right_w2_controller', Namespace: /robot/right_joint_effort_controller/joints/right_w2_controller
[INFO] [WallTime: 1455549597.781695] [2.022000] Controller Spawner: Loaded controllers: left_joint_position_controller, right_joint_position_controller, head_position_controller, left_joint_velocity_controller, right_joint_velocity_controller, left_joint_effort_controller, right_joint_effort_controller
[ INFO] [1455549630.368986084, 34.260000000]: Simulator is loaded and started successfully
[ INFO] [1455549630.378603960, 34.261000000]: Robot is disabled
[ INFO] [1455549630.378940511, 34.261000000]: Gravity compensation was turned off
[ERROR] [1455549970.115578834, 366.801000000]: Could not stop controller 'left_joint_velocity_controller' since it is not running
[ERROR] [1455549970.116366375, 366.802000000]: Failed to switch controllers
[ERROR] [1455549974.722494880, 371.253000000]: Could not stop controller 'right_joint_velocity_controller' since it is not running
[ERROR] [1455549974.722635254, 371.253000000]: Failed to switch controllers

Controller spawner says it loaded the controllers, but when I'm trying to use wobbler, or tuck_arms the last two error messages repeat.

I do remember, that this simulation was working before, maybe dependencies changed or some configuration error happened, but the messages have some conflict between them (loaded, but not running controllers?)

Note: Also sometimes (not every time!) I get parts of the robot falling apart in gazebo with messages like:

[INFO] [WallTime: 1455548845.702339] [0.000000] Controller Spawner: Waiting for service controller_manager/load_controller
Unknown property der
Unknown property der
QMetaObject::connectSlotsByName: No matching signal for on_left_shoulder_pressed()
QMetaObject::connectSlotsByName: No matching signal for on_left_shoulder_released()
QMetaObject::connectSlotsByName: No matching signal for on_right_shoulder_pressed()
QMetaObject::connectSlotsByName: No matching signal for on_right_shoulder_released()
Error [Param.cc:181] Unable to set value [1,0471975511965976] for key[horizontal_fov]
Error [Param.cc:181] Unable to set value [0,100000001] for key[near]
Warning [parser_urdf.cc:1010] multiple inconsistent <self_collide> exists due to fixed joint reduction overwriting previous value [true] with [false].

or

[INFO] [WallTime: 1455548846.458276] [0.755000] temporary hack to **fix** the -J joint position option (issue #93), sleeping for 1 second to avoid race condition.
[ INFO] [1455548846.514682640, 0.755000000]: Camera Plugin (ns = /cameras)  <tf_prefix_>, set to "/cameras"
[ INFO] [1455548846.585618371, 0.755000000]: Camera Plugin (ns = /cameras)  <tf_prefix_>, set to "/cameras"
[ INFO] [1455548846.639640254, 0.755000000]: Camera Plugin (ns = /cameras)  <tf_prefix_>, set to "/cameras"
[ INFO] [1455548846.949493719]: GazeboRosVideo (gzclient, ns = /) has started!
Exception [ODELink.cc:386] Setting custom link baxter::left_lower_elbowmass to zero!

Error [World.cc:1666] Loading model from factory message failed
[INFO] [WallTime: 1455548847.460114] [1.168000] Calling service /gazebo/set_model_configuration
[INFO] [WallTime: 1455548847.462946] [1.172000] Set model configuration status: SetModelConfiguration: success
[ INFO] [1455548847.590697698, 1.295000000]: GazeboRosVideo (gzserver, ns = /) has started!
[urdf_spawner-5] process has finished cleanly
log file: /home/baxterws/.ros/log/d18b9e80-d3f5-11e5-873d-7824af346e2e/urdf_spawner-5*.log
[WARN] [WallTime: 1455548875.920646] [29.422000] Controller Spawner couldn't find the expected controller_manager ROS interface.
[WARN] [WallTime: 1455548875.942996] [29.445000] Controller Spawner couldn't find the expected controller_manager ROS interface.
[robot/controller_spawner-9] process has finished cleanly
log file: /home/baxterws/.ros/log/d18b9e80-d3f5-11e5-873d-7824af346e2e/robot-controller_spawner-9*.log
[robot/controller_spawner_stopped-10] process has finished cleanly
log file: /home/baxterws/.ros/log/d18b9e80-d3f5-11e5-873d-7824af346e2e/robot-controller_spawner_stopped-10*.log
[ INFO] [1455548881.734463309, 35.206000000]: Simulator is loaded and started successfully
Warning [Publisher.cc:134] Queue limit reached for topic /gazebo/default/pose/local/info, deleting message. This warning is printed only once.

These might be independent issues, but I can't find direction where to look for clues.

The commands I run:

[baxter - http://localhost:11311] baxterws@baxterws-operator:~/ros_ws$ rosrun baxter_tools enable_robot.py -e
[INFO] [WallTime: 1455549666.909481] [70.466000] Robot Enabled
[baxter - http://localhost:11311] baxterws@baxterws-operator:~/ros_ws$ rosrun baxter_tools tuck_arms.py -t
[INFO] [WallTime: 1455549681.931965] [0.000000] Tucking arms
Traceback (most recent call last):
  File "/home/baxterws/ros_ws/src/baxter_tools/scripts/tuck_arms.py", line 266, in <module>
    main()
  File "/home/baxterws/ros_ws/src/baxter_tools/scripts/tuck_arms.py", line 260, in main
    tucker = Tuck(tuck)
  File "/home/baxterws/ros_ws/src/baxter_tools/scripts/tuck_arms.py", line 57, in __init__
    'left': baxter_interface.Limb('left'),
  File "/home/baxterws/ros_ws/src/baxter_interface/src/baxter_interface/limb.py", line 121, in __init__
    timeout_msg=err_msg)
  File "/home/baxterws/ros_ws/src/baxter_interface/src/baxter_dataflow/wait_for.py", line 55, in wait_for
    raise OSError(errno.ETIMEDOUT, timeout_msg)
OSError: [Errno 110] Left limb init failed to get current joint_states from robot/joint_states

[baxter - http://localhost:11311] baxterws@baxterws-operator:~/ros_ws$ rostopic hz robot/joint_states
subscribed to [/robot/joint_states]
average rate: 50.000
    min: 0.019s max: 0.021s std dev: 0.00050s window: 49
average rate: 50.000
    min: 0.012s max: 0.028s std dev: 0.00125s window: 98
average rate: 50.000
    min: 0.012s max: 0.028s std dev: 0.00108s window: 147
^Caverage rate: 50.014
    min: 0.012s max: 0.028s std dev: 0.00101s window: 175

[baxter - http://localhost:11311] baxterws@baxterws-operator:~/ros_ws$ rostopic echo robot/joint_states

---
header: 
  seq: 12220
  stamp: 
    secs: 245
    nsecs: 779000000
  frame_id: ''
name: ['head_pan', 'left_e0', 'left_e1', 'left_s0', 'left_s1', 'left_w0', 'left_w1', 'left_w2', 'right_e0', 'right_e1', 'right_s0', 'right_s1', 'right_w0', 'right_w1', 'right_w2']
position: [-0.012458599015063854, 0.013087865650309283, 0.5214991063283057, 0.1849393878969705, 1.031804682474478, -0.17780065013399504, -0.057332130027723416, -0.012499985633008137, 0.01782652946334462, 0.5033388615200369, -0.9082738979939275, 1.047003724560148, -0.9197235288933481, -0.05602882311736135, -0.02588254594208017]
velocity: [0.0024714338851410484, 0.00045443402907291213, -0.0006092397220345934, 0.003311871112331409, -0.0003337174578951593, 0.0016997448839176535, 0.0013185028807170987, 0.0004332011476928019, -0.0003476372584358562, 0.001129744222407989, -0.0013019436282886033, -0.00015323960617145597, -0.002788218175726901, -0.0009309409860963384, -0.0001679991253535644]
effort: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

---

[baxter - http://localhost:11311] baxterws@baxterws-operator:~/ros_ws$ rosrun baxter_examples joint_velocity_wobbler.py 
Initializing node... 
Getting robot state... 
Enabling robot... 
[INFO] [WallTime: 1455549969.971199] [366.657000] Robot Enabled
Moving to neutral pose...
^C
Exiting example...
Moving to neutral pose...

gazebo says:

[ERROR] [1455549970.115578834, 366.801000000]: Could not stop controller 'left_joint_velocity_controller' since it is not running
[ERROR] [1455549970.116366375, 366.802000000]: Failed to switch controllers
[ERROR] [1455549974.722494880, 371.253000000]: Could not stop controller 'right_joint_velocity_controller' since it is not running
[ERROR] [1455549974.722635254, 371.253000000]: Failed to switch controllers
[baxter - http://localhost:11311] baxterws@baxterws-operator:~/ros_ws$ env | grep ROS
ROS_ROOT=/opt/ros/indigo/share/ros
ROS_PACKAGE_PATH=/home/baxterws/ros_ws/src:/opt/ros/indigo/share:/opt/ros/indigo/stacks
ROS_MASTER_URI=http://localhost:11311
ROS_TEST_RESULTS_DIR=/home/baxterws/ros_ws/build/test_results
ROSLISP_PACKAGE_DIRECTORIES=/home/baxterws/ros_ws/devel/share/common-lisp
__ROS_PROMPT=1
ROS_DISTRO=indigo
ROS_IP=192.168.6.127
ROS_ETC_DIR=/opt/ros/indigo/etc/ros

Both roslaunch working in paralell and gazebo having timing and racecondition issues gives the whole system a pretty you-can-never-reproduce-this-error taste. :)

I do have the same problem on a different machine, where the ros and osrf packages were kept up-to date via apt, but the sources weren't touched in 6 months, and gazebo 2.2.6 shows the same simptoms:

Could not stop controller 'left_joint_velocity_controller' since it is not running.

Any advice?

rethink-imcmahon commented 8 years ago

It's going to be difficult to debug this without being able to see your exact machine. Several of the issues you have mentioned were warnings that I cleaned up in RSDK 1.2.

The errors might very well be issues of ROS sources changing their API's out from underneath your baxter_simulator. You might try using the ros_controls and ros_controllers source code for indigo-devel, and rolling them back to the date where you found the simulation last worked: https://github.com/ros-controls/ros_controllers/tree/indigo-devel