luxai-qtrobot / QA

Virtual repository for Questions & Answers system
http://luxai-qtrobot.github.io
5 stars 0 forks source link

Motor node is crashing #47

Closed TechnoX closed 2 years ago

TechnoX commented 3 years ago

After a while the qt_motor node dies. When I write rosnode info /qt_motor I get the following response:

ERROR: Communication with node[http://192.168.100.1:41301/] failed!

When I open the start_qt_motor.log file I see the following:

#############################################
 QT Robot autostart version <1.0>
 Running [start_qt_motor]
 Don Jun  3 18:24:17 CEST 2021
#############################################

[Autostart] Preparing ROS environment [OK]
[Autostart] Waiting for ROS node /rosout
[Autostart] Waiting for TCPIP port 11311
[Autostart] TCPIP port 11311 [OK]
[Autostart] ROS node /rosout [ok]
... logging to /home/qtrobot/.ros/log/26ae17de-c488-11eb-acb6-b827eb12c4d0/roslaunch-QTRD000185-1691.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
]2;/opt/ros/kinetic/share/qt_motor/launch/qt_motor.launch
started roslaunch server http://192.168.100.1:43157/

SUMMARY
========

CLEAR PARAMETERS
 * /controller/
 * /qt_motor/

PARAMETERS
 * /herkulex_control/loop_frequency: 10
 * /herkulex_control/motors/HeadPitch/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/HeadPitch/callibration/offset: 1.63
 * /herkulex_control/motors/HeadPitch/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/HeadPitch/id: 10
 * /herkulex_control/motors/HeadPitch/index: 1
 * /herkulex_control/motors/HeadPitch/overload/period: 70
 * /herkulex_control/motors/HeadPitch/overload/threshold: 1000
 * /herkulex_control/motors/HeadPitch/part: head
 * /herkulex_control/motors/HeadPitch/pid/dead_zone: 0.0
 * /herkulex_control/motors/HeadPitch/pid/kd: 150
 * /herkulex_control/motors/HeadPitch/pid/ki: 0
 * /herkulex_control/motors/HeadPitch/pid/kp: 175
 * /herkulex_control/motors/HeadPitch/pid/sat_offset: 254
 * /herkulex_control/motors/HeadPitch/pid/sat_slope: 6000
 * /herkulex_control/motors/HeadPitch/pos/home: 0.0
 * /herkulex_control/motors/HeadPitch/pos/max: 21.63
 * /herkulex_control/motors/HeadPitch/pos/min: -13.37
 * /herkulex_control/motors/HeadPitch/pos/park: 23
 * /herkulex_control/motors/HeadPitch/velocity/default: 5
 * /herkulex_control/motors/HeadPitch/velocity/max: 30
 * /herkulex_control/motors/HeadYaw/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/HeadYaw/callibration/offset: -3.58
 * /herkulex_control/motors/HeadYaw/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/HeadYaw/id: 5
 * /herkulex_control/motors/HeadYaw/index: 0
 * /herkulex_control/motors/HeadYaw/overload/period: 20
 * /herkulex_control/motors/HeadYaw/overload/threshold: 400
 * /herkulex_control/motors/HeadYaw/part: head
 * /herkulex_control/motors/HeadYaw/pid/dead_zone: 0.0
 * /herkulex_control/motors/HeadYaw/pid/kd: 150
 * /herkulex_control/motors/HeadYaw/pid/ki: 0
 * /herkulex_control/motors/HeadYaw/pid/kp: 175
 * /herkulex_control/motors/HeadYaw/pid/sat_offset: 254
 * /herkulex_control/motors/HeadYaw/pid/sat_slope: 3000
 * /herkulex_control/motors/HeadYaw/pos/home: 0
 * /herkulex_control/motors/HeadYaw/pos/max: 54.42
 * /herkulex_control/motors/HeadYaw/pos/min: -61.58
 * /herkulex_control/motors/HeadYaw/pos/park: 0
 * /herkulex_control/motors/HeadYaw/velocity/default: 5
 * /herkulex_control/motors/HeadYaw/velocity/max: 100
 * /herkulex_control/motors/LeftElbowRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/LeftElbowRoll/callibration/offset: -2.27
 * /herkulex_control/motors/LeftElbowRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/LeftElbowRoll/id: 8
 * /herkulex_control/motors/LeftElbowRoll/index: 7
 * /herkulex_control/motors/LeftElbowRoll/overload/period: 20
 * /herkulex_control/motors/LeftElbowRoll/overload/threshold: 400
 * /herkulex_control/motors/LeftElbowRoll/part: left_arm
 * /herkulex_control/motors/LeftElbowRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/LeftElbowRoll/pid/kd: 100
 * /herkulex_control/motors/LeftElbowRoll/pid/ki: 0
 * /herkulex_control/motors/LeftElbowRoll/pid/kp: 100
 * /herkulex_control/motors/LeftElbowRoll/pid/sat_offset: 250
 * /herkulex_control/motors/LeftElbowRoll/pid/sat_slope: 700
 * /herkulex_control/motors/LeftElbowRoll/pos/home: -35.0
 * /herkulex_control/motors/LeftElbowRoll/pos/max: -7.27
 * /herkulex_control/motors/LeftElbowRoll/pos/min: -82.27
 * /herkulex_control/motors/LeftElbowRoll/pos/park: -10.0
 * /herkulex_control/motors/LeftElbowRoll/velocity/default: 15
 * /herkulex_control/motors/LeftElbowRoll/velocity/max: 100
 * /herkulex_control/motors/LeftShoulderPitch/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/LeftShoulderPitch/callibration/offset: 2.6
 * /herkulex_control/motors/LeftShoulderPitch/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/LeftShoulderPitch/id: 6
 * /herkulex_control/motors/LeftShoulderPitch/index: 5
 * /herkulex_control/motors/LeftShoulderPitch/overload/period: 20
 * /herkulex_control/motors/LeftShoulderPitch/overload/threshold: 600
 * /herkulex_control/motors/LeftShoulderPitch/part: left_arm
 * /herkulex_control/motors/LeftShoulderPitch/pid/dead_zone: 0.0
 * /herkulex_control/motors/LeftShoulderPitch/pid/kd: 100
 * /herkulex_control/motors/LeftShoulderPitch/pid/ki: 0
 * /herkulex_control/motors/LeftShoulderPitch/pid/kp: 100
 * /herkulex_control/motors/LeftShoulderPitch/pid/sat_offset: 250
 * /herkulex_control/motors/LeftShoulderPitch/pid/sat_slope: 800
 * /herkulex_control/motors/LeftShoulderPitch/pos/home: 90.0
 * /herkulex_control/motors/LeftShoulderPitch/pos/max: 140.6
 * /herkulex_control/motors/LeftShoulderPitch/pos/min: -135.4
 * /herkulex_control/motors/LeftShoulderPitch/pos/park: 90.0
 * /herkulex_control/motors/LeftShoulderPitch/velocity/default: 15
 * /herkulex_control/motors/LeftShoulderPitch/velocity/max: 100
 * /herkulex_control/motors/LeftShoulderRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/LeftShoulderRoll/callibration/offset: 1.63
 * /herkulex_control/motors/LeftShoulderRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/LeftShoulderRoll/id: 7
 * /herkulex_control/motors/LeftShoulderRoll/index: 6
 * /herkulex_control/motors/LeftShoulderRoll/overload/period: 20
 * /herkulex_control/motors/LeftShoulderRoll/overload/threshold: 600
 * /herkulex_control/motors/LeftShoulderRoll/part: left_arm
 * /herkulex_control/motors/LeftShoulderRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/LeftShoulderRoll/pid/kd: 100
 * /herkulex_control/motors/LeftShoulderRoll/pid/ki: 0
 * /herkulex_control/motors/LeftShoulderRoll/pid/kp: 100
 * /herkulex_control/motors/LeftShoulderRoll/pid/sat_offset: 250
 * /herkulex_control/motors/LeftShoulderRoll/pid/sat_slope: 800
 * /herkulex_control/motors/LeftShoulderRoll/pos/home: -55.0
 * /herkulex_control/motors/LeftShoulderRoll/pos/max: -13.37
 * /herkulex_control/motors/LeftShoulderRoll/pos/min: -78.37
 * /herkulex_control/motors/LeftShoulderRoll/pos/park: -65.0
 * /herkulex_control/motors/LeftShoulderRoll/velocity/default: 15
 * /herkulex_control/motors/LeftShoulderRoll/velocity/max: 100
 * /herkulex_control/motors/RightElbowRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/RightElbowRoll/callibration/offset: -9.75
 * /herkulex_control/motors/RightElbowRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/RightElbowRoll/id: 2
 * /herkulex_control/motors/RightElbowRoll/index: 4
 * /herkulex_control/motors/RightElbowRoll/overload/period: 20
 * /herkulex_control/motors/RightElbowRoll/overload/threshold: 400
 * /herkulex_control/motors/RightElbowRoll/part: right_arm
 * /herkulex_control/motors/RightElbowRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/RightElbowRoll/pid/kd: 100
 * /herkulex_control/motors/RightElbowRoll/pid/ki: 0
 * /herkulex_control/motors/RightElbowRoll/pid/kp: 100
 * /herkulex_control/motors/RightElbowRoll/pid/sat_offset: 250
 * /herkulex_control/motors/RightElbowRoll/pid/sat_slope: 700
 * /herkulex_control/motors/RightElbowRoll/pos/home: -35.0
 * /herkulex_control/motors/RightElbowRoll/pos/max: -14.75
 * /herkulex_control/motors/RightElbowRoll/pos/min: -89.75
 * /herkulex_control/motors/RightElbowRoll/pos/park: -10.0
 * /herkulex_control/motors/RightElbowRoll/velocity/default: 15
 * /herkulex_control/motors/RightElbowRoll/velocity/max: 100
 * /herkulex_control/motors/RightShoulderPitch/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/RightShoulderPitch/callibration/offset: -4.23
 * /herkulex_control/motors/RightShoulderPitch/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/RightShoulderPitch/id: 4
 * /herkulex_control/motors/RightShoulderPitch/index: 2
 * /herkulex_control/motors/RightShoulderPitch/overload/period: 20
 * /herkulex_control/motors/RightShoulderPitch/overload/threshold: 600
 * /herkulex_control/motors/RightShoulderPitch/part: right_arm
 * /herkulex_control/motors/RightShoulderPitch/pid/dead_zone: 0.0
 * /herkulex_control/motors/RightShoulderPitch/pid/kd: 100
 * /herkulex_control/motors/RightShoulderPitch/pid/ki: 0
 * /herkulex_control/motors/RightShoulderPitch/pid/kp: 100
 * /herkulex_control/motors/RightShoulderPitch/pid/sat_offset: 250
 * /herkulex_control/motors/RightShoulderPitch/pid/sat_slope: 800
 * /herkulex_control/motors/RightShoulderPitch/pos/home: -90.0
 * /herkulex_control/motors/RightShoulderPitch/pos/max: 133.77
 * /herkulex_control/motors/RightShoulderPitch/pos/min: -142.23
 * /herkulex_control/motors/RightShoulderPitch/pos/park: -90.0
 * /herkulex_control/motors/RightShoulderPitch/velocity/default: 15
 * /herkulex_control/motors/RightShoulderPitch/velocity/max: 100
 * /herkulex_control/motors/RightShoulderRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/RightShoulderRoll/callibration/offset: 11.38
 * /herkulex_control/motors/RightShoulderRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/RightShoulderRoll/id: 3
 * /herkulex_control/motors/RightShoulderRoll/index: 3
 * /herkulex_control/motors/RightShoulderRoll/overload/period: 20
 * /herkulex_control/motors/RightShoulderRoll/overload/threshold: 600
 * /herkulex_control/motors/RightShoulderRoll/part: right_arm
 * /herkulex_control/motors/RightShoulderRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/RightShoulderRoll/pid/kd: 100
 * /herkulex_control/motors/RightShoulderRoll/pid/ki: 0
 * /herkulex_control/motors/RightShoulderRoll/pid/kp: 100
 * /herkulex_control/motors/RightShoulderRoll/pid/sat_offset: 250
 * /herkulex_control/motors/RightShoulderRoll/pid/sat_slope: 800
 * /herkulex_control/motors/RightShoulderRoll/pos/home: -55.0
 * /herkulex_control/motors/RightShoulderRoll/pos/max: -3.62
 * /herkulex_control/motors/RightShoulderRoll/pos/min: -68.62
 * /herkulex_control/motors/RightShoulderRoll/pos/park: -65.0
 * /herkulex_control/motors/RightShoulderRoll/velocity/default: 15
 * /herkulex_control/motors/RightShoulderRoll/velocity/max: 100
 * /herkulex_control/serial_baudrate: 115200
 * /herkulex_control/serial_interface: /dev/serial0
 * /herkulex_control/serial_read_timeout: 10
 * /herkulex_control/status_frequency: 1
 * /herkulex_control/unit_radian: False
 * /qt_robot/HeadPitch_position/joint: HeadPitch
 * /qt_robot/HeadPitch_position/pid/d: 0.0
 * /qt_robot/HeadPitch_position/pid/i: 1.0
 * /qt_robot/HeadPitch_position/pid/p: 1.0
 * /qt_robot/HeadPitch_position/type: position_controll...
 * /qt_robot/HeadYaw_position/joint: HeadYaw
 * /qt_robot/HeadYaw_position/pid/d: 0.0
 * /qt_robot/HeadYaw_position/pid/i: 1.0
 * /qt_robot/HeadYaw_position/pid/p: 1.0
 * /qt_robot/HeadYaw_position/type: position_controll...
 * /qt_robot/LeftElbowRoll_position/joint: LeftElbowRoll
 * /qt_robot/LeftElbowRoll_position/pid/d: 0.0
 * /qt_robot/LeftElbowRoll_position/pid/i: 1.0
 * /qt_robot/LeftElbowRoll_position/pid/p: 1.0
 * /qt_robot/LeftElbowRoll_position/type: position_controll...
 * /qt_robot/LeftShoulderPitch_position/joint: LeftShoulderPitch
 * /qt_robot/LeftShoulderPitch_position/pid/d: 0.0
 * /qt_robot/LeftShoulderPitch_position/pid/i: 1.0
 * /qt_robot/LeftShoulderPitch_position/pid/p: 1.0
 * /qt_robot/LeftShoulderPitch_position/type: position_controll...
 * /qt_robot/LeftShoulderRoll_position/joint: LeftShoulderRoll
 * /qt_robot/LeftShoulderRoll_position/pid/d: 0.0
 * /qt_robot/LeftShoulderRoll_position/pid/i: 1.0
 * /qt_robot/LeftShoulderRoll_position/pid/p: 1.0
 * /qt_robot/LeftShoulderRoll_position/type: position_controll...
 * /qt_robot/RightElbowRoll_position/joint: RightElbowRoll
 * /qt_robot/RightElbowRoll_position/pid/d: 0.0
 * /qt_robot/RightElbowRoll_position/pid/i: 1.0
 * /qt_robot/RightElbowRoll_position/pid/p: 1.0
 * /qt_robot/RightElbowRoll_position/type: position_controll...
 * /qt_robot/RightShoulderPitch_position/joint: RightShoulderPitch
 * /qt_robot/RightShoulderPitch_position/pid/d: 0.0
 * /qt_robot/RightShoulderPitch_position/pid/i: 1.0
 * /qt_robot/RightShoulderPitch_position/pid/p: 1.0
 * /qt_robot/RightShoulderPitch_position/type: position_controll...
 * /qt_robot/RightShoulderterminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error> >'
  what():  boost::thread_resource_error: Resource temporarily unavailable
[ INFO] [1622737475.409542387]:      Found HeadPitch index:1
[ INFO] [1622737475.473454262]:      Found HeadYaw index:0
[ INFO] [1622737475.541547439]:      Found LeftElbowRoll index:7
[ INFO] [1622737475.599535773]:      Found LeftShoulderPitch index:5
[ INFO] [1622737475.655743221]:      Found LeftShoulderRoll index:6
[ INFO] [1622737475.716459210]:      Found RightElbowRoll index:4
[ INFO] [1622737475.788136293]:      Found RightShoulderPitch index:2
[ INFO] [1622737475.850342648]:      Found RightShoulderRoll index:3
[ INFO] [1622737475.908729418]: Loaded 8 motors configuration parameters!
[ INFO] [1622737475.926823585]: Opening herkulex motors driver... (baudrate: 115200)
[ INFO] [1622737476.427611449]: Rebooting motors...
[ INFO] [1622737479.456511657]: *** Setting temp/voltage limits ***
[ INFO] [1622737479.462211240]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (HeadYaw)
[ INFO] [1622737479.465142386]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (HeadPitch)
[ INFO] [1622737479.468107594]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (RightShoulderPitch)
[ INFO] [1622737479.471136136]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (RightShoulderRoll)
[ INFO] [1622737479.474039209]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (RightElbowRoll)
[ INFO] [1622737479.476986500]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (LeftShoulderPitch)
[ INFO] [1622737479.479893740]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (LeftShoulderRoll)
[ INFO] [1622737479.482835823]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (LeftElbowRoll)
[ INFO] [1622737479.482951500]: ***********************************
[ INFO] [1622737479.483025615]: *** Setting callibration params ***
[ INFO] [1622737479.483282334]: *** Setting PID and compliancy ***
[ INFO] [1622737479.498115302]: *** Setting overload detection values ***
[ INFO] [1622737479.518339000]: *** Turning all motors torque ***
[ INFO] [1622737482.534756916]: *** Clearing all motor status ***
[ INFO] [1622737482.534953582]: *** Setting joint limits ***
[ INFO] [1622737482.535107645]: *** Reading joints current postion ***
[ INFO] [1622737482.554252957]: Pos motor HeadYaw : 0.00
[ INFO] [1622737482.557410562]: Pos motor HeadPitch : 0.60
[ INFO] [1622737482.560591864]: Pos motor RightShoulderPitch : -89.90
[ INFO] [1622737482.563771760]: Pos motor RightShoulderRoll : -58.60
[ INFO] [1622737482.566855145]: Pos motor RightElbowRoll : -35.20
[ INFO] [1622737482.569824832]: Pos motor LeftShoulderPitch : 89.60
[ INFO] [1622737482.572804312]: Pos motor LeftShoulderRoll : -58.60
[ INFO] [1622737482.575847489]: Pos motor LeftElbowRoll : -34.50
[ INFO] [1622737482.576446239]: *** Staring controller main loop ***
[ INFO] [1622737492.116603266]: QTMotorsController: starting
[ INFO] [1622737492.116774048]: QTGestureController: starting
[ WARN] [1622737731.095011128]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1622737731.095211597]: Cannot read behavior of motor LeftShoulderRoll!
[ WARN] [1622738739.685093611]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1622738739.685298591]: Cannot read behavior of motor HeadYaw!
================================================================================REQUIRED process [qt_motor-1] has died!
process has died [pid 1846, exit code -6, cmd /opt/ros/kinetic/lib/qt_motor/qt_motor joint_states:=/qt_robot/joints/state __name:=qt_motor __log:=/home/qtrobot/.ros/log/26ae17de-c488-11eb-acb6-b827eb12c4d0/qt_motor-1.log].
log file: /home/qtrobot/.ros/log/26ae17de-c488-11eb-acb6-b827eb12c4d0/qt_motor-1*.log
Initiating shutdown!
================================================================================
[WARN] [1622739173.457589]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[INFO] [1622737491.353701]: Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [1622737491.361208]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1622737491.367837]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1622737491.374550]: Loading controller: /qt_robot/head_position
[INFO] [1622737491.517477]: Loading controller: /qt_robot/right_arm_position
[INFO] [1622737491.617533]: Loading controller: /qt_robot/left_arm_position
[INFO] [1622737491.717705]: Loading controller: /qt_robot/joints_state
[INFO] [1622737491.818703]: Loading controller: /qt_robot/motors
[INFO] [1622737491.917941]: Loading controller: /qt_robot/gesture
[INFO] [1622737492.043019]: Controller Spawner: Loaded controllers: /qt_robot/head_position, /qt_robot/right_arm_position, /qt_robot/left_arm_position, /qt_robot/joints_state, /qt_robot/motors, /qt_robot/gesture
[INFO] [1622737492.117988]: Started controllers: /qt_robot/head_position, /qt_robot/right_arm_position, /qt_robot/left_arm_position, /qt_robot/joints_state, /qt_robot/motors, /qt_robot/gesture
[INFO] [1622739173.446360]: Shutting down spawner. Stopping and unloading controllers...
[INFO] [1622739173.447996]: Stopping all controllers...
Roll_position/joint: RightShoulderRoll
 * /qt_robot/RightShoulderRoll_position/pid/d: 0.0
 * /qt_robot/RightShoulderRoll_position/pid/i: 1.0
 * /qt_robot/RightShoulderRoll_position/pid/p: 1.0
 * /qt_robot/RightShoulderRoll_position/type: position_controll...
 * /qt_robot/gesture/adjust_mainloop_frequency: False
 * /qt_robot/gesture/gestures_path: /home/qtrobot/rob...
 * /qt_robot/gesture/type: qt_gesture_contro...
 * /qt_robot/head_controller/joints: ['HeadYaw', 'Head...
 * /qt_robot/head_controller/type: position_controll...
 * /qt_robot/head_position/joints: ['HeadYaw', 'Head...
 * /qt_robot/head_position/type: position_controll...
 * /qt_robot/joints_state/publish_rate: 10
 * /qt_robot/joints_state/type: joint_state_contr...
 * /qt_robot/left_arm_controller/joints: ['LeftShoulderPit...
 * /qt_robot/left_arm_controller/type: position_controll...
 * /qt_robot/left_arm_position/joints: ['LeftShoulderPit...
 * /qt_robot/left_arm_position/type: position_controll...
 * /qt_robot/motors/motors_state_publish_rate: 1
 * /qt_robot/motors/type: qt_motors_control...
 * /qt_robot/right_arm_controller/joints: ['RightShoulderPi...
 * /qt_robot/right_arm_controller/type: position_controll...
 * /qt_robot/right_arm_position/joints: ['RightShoulderPi...
 * /qt_robot/right_arm_position/type: position_controll...
 * /rosdistro: kinetic
 * /rosversion: 1.12.17

NODES
  /
    controller (controller_manager/spawner)
    qt_motor (qt_motor/qt_motor)

ROS_MASTER_URI=http://192.168.100.1:11311
]2;/opt/ros/kinetic/share/qt_motor/launch/qt_motor.launch http://192.168.100.1:11311
process[qt_motor-1]: started with pid [1846]
process[controller-2]: started with pid [1849]
[controller-2] killing on exit
[qt_motor-1] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

My suspicion is that the node is not thread-safe. But I have no proof for this. However it happens sporadically, I haven't seen a pattern. And that is usually the case with race conditions or thread bugs.

I have one node calling for the setVelocity service, and another node calling for the gesture services (when the above happened it was the service /qt_robot/gesture/stop and qt_robot/motors/setVelocity (for HeadYaw). May these two collide somehow?

apaikan commented 3 years ago

Hi @TechnoX This is a bit strange since we have not experienced any crash of qt_motor. It might be the related to unsafe thread calls and requires performing more in depth debugging.

Just to clarify, qt_robot/motors/setVelocity should not and has not any effect on the speed of the gesture controller. QTrobot gesture controller plugin, directly send low-level commands the the motors and does not use ROS position controller.

The setVelocity just set the default velocity of motor position controller and will affect the next published commands to joint position controller (e.g. /qt_robot/head_position/command) and must not be used in the same time of playing gesture. It's mostly an alternative solution to set the motors default speed (unrelated to the gesture) to the qt_motor config file. As you may know, it is not neither a standard interface of ros_control package.

If you can share your scenario, I may be able to propose better solution.

TechnoX commented 3 years ago

Thanks for the quick reply! I can add that it has happened 4-5 times for us, so it is not frequent, but neither very unfrequent. Just tell me if I can help with the debugging somehow. If I should send you any more logs or test to swap the qt_motor node to another one with more debugging capabilities.

The scenario is as follows:

  1. I have one node for controlling the neck (HeadYaw and HeadPitch), this node subscribes on face detections (from another image processing node) and try to keep the face in the image at all times. Such that the robot looks at you. This node is sending messages to /qt_robot/head_position/command. And to not get overshoot I have implemented a proportional control algorithm that sets the speed of the motors with service calls to /qt_robot/motors/setVelocity

(PS. Unrelated question, is it possible to set different speed of different joints in the same service call? I see that the first argument is an array, but all joints are set to the same speed as I can see. It would be nice if also the second argument can be an array with speeds. Currently I'm doing two consecutive service calls:

self.velocity_service(['HeadYaw'], yaw_speed)
self.velocity_service(['HeadPitch'], pitch_speed)

DS.)

For the sake of completeness I can also add that this node subscribes on "/qt_robot/joints/state" to get the current positions of the joints. It might be a thread bug when trying to access the position at the same time as it is written to them?

  1. I have another node that is responsible for gestures when the robot is talking. This node subscribes on a topic with intents and process them to decide appropriate gestures. This node sends service calls to /qt_robot/gesture/stop, qt_robot/gesture/play and qt_robot/motors/home. It does also send messages on the /qt_robot/gesture/play topic.

Most of our gestures are only affecting the arms. But some of them might also control the head. It's obvious that you can't both follow a face with the head and at the same time play for example a nodding gesture with the neck, since the motors can't be at two positions simultaneous.

But when I did my experimenting on this I saw that the gesture service calls (and topics) took precedence over the /qt_robot/head_position/command topic. I.e. I can publish a message for the head to rotate to HeadYaw 30 degrees and while it's moving I issue a call to /qt_robot/gesture/play and it will play that gesture. This is the desired behavior as well. The robot should track the face and when having a nodding intent it can nod and when done with the gesture it will continue to track the face. Perfect!

But it seems like this procedure very seldom generates a crash in the motor node ... This should be possible to fix I would say.

apaikan commented 3 years ago

Thanks @TechnoX. everything is more clear now.

In summary you are :

Anyway beside the crash bug you have reported I see few other issues with this setup:

  1. As I mentioned before, the setVelocity service was not meant to be called continuously to adjust motor's position controller speed at run-time. so we need to debug it more deeply its behavior.
  2. As you said, you probably need to slow down the tracking to cope with the overshot in the control loop.

So here are my suggested solutions:

[Solution 1] Using trajectory controller

The QTrobot position controller/joint state publisher are based on standard ros_control package. That means you can use already existing joint_trajectory_controller instead of your custom velocity controller. The joint_trajectory_controller can leverage QTrobot position controller and has internal PID configuration.

All you need is to run qt_motor in advance mode (roslaunch qt_motor qt_motor_advanced.launch). The advanced mode change some robot's configuration and behavior so that:

You can use rqt_joint_trajectory_controller graphical interface to test.

[Solution 2] Using inverse kinematic and cartesian space

This is my preferred solution: to command the head in cartesian space and implement a gaze controller for QTrobot. Before getting in how to do it, let me explain how it usually works:

  1. The detected face pos (pixels) in image frame is converted to 3D pos in camera frame. A simple way to do that is to use Nuitrack skeleton tracking data for the head pos. Alternatively one can use the point cloud/disparity map from realness to get the 3D pose. It can also be done with 2D image and calibrated camera data.
  2. The 3D pose in camera frame then can be easily converted to the robot frame using QTrobot URFD model.
  3. The 3D pose is given to the gaze controller which use the inverse kinematic solver of the head to look at a 3D pose in the space by sending even a single command to the motors which control the head.

This solution should offer very fast and robust face tracking using any robot. So what's missing and what should be done:

If you are eager to contribute to this, let me know to provide you some guideline and links.

Cheers!

TechnoX commented 3 years ago

Thank you for the elaborated answer!

First I can tell you that the qt_motor_advanced.launch gave me a crash (I was not calling any service or gestures, it just died):

[SKIPPED FIRST LINES]
[ INFO] [1623280041.702589102]: Pos motor LeftShoulderPitch : 89.60
[ INFO] [1623280041.705503022]: Pos motor LeftShoulderRoll : -57.00
[ INFO] [1623280041.708391369]: Pos motor LeftElbowRoll : -34.80
[ INFO] [1623280041.708761729]: *** Staring controller main loop ***
[INFO] [1623280050.534930]: Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [1623280050.545169]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1623280050.554754]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1623280050.562179]: Loading controller: /qt_robot/head_controller
[INFO] [1623280050.815840]: Loading controller: /qt_robot/right_arm_controller
[INFO] [1623280050.988470]: Loading controller: /qt_robot/left_arm_controller
[INFO] [1623280051.168370]: Loading controller: /qt_robot/joints_state
[INFO] [1623280051.232487]: Loading controller: /qt_robot/motors
[INFO] [1623280051.305729]: Loading controller: /qt_robot/gesture
[INFO] [1623280051.405697]: Controller Spawner: Loaded controllers: /qt_robot/head_controller, /qt_robot/right_arm_controller, /qt_robot/left_arm_controller, /qt_robot/joints_state, /qt_robot/motors, /qt_robot/gesture
[ INFO] [1623280051.443088691]: QTMotorsController: starting
[ INFO] [1623280051.443707745]: QTGestureController: starting
[INFO] [1623280051.445643]: Started controllers: /qt_robot/head_controller, /qt_robot/right_arm_controller, /qt_robot/left_arm_controller, /qt_robot/joints_state, /qt_robot/motors, /qt_robot/gesture
terminate called after throwing an instance of 'std::runtime_error'
  what():  Time is out of dual 32-bit range
================================================================================REQUIRED process [qt_motor-1] has died!
process has died [pid 6632, exit code -6, cmd /opt/ros/kinetic/lib/qt_motor/qt_motor joint_states:=/qt_robot/joints/state __name:=qt_motor __log:=/home/qtrobot/.ros/log/4ee520d6-c950-11eb-9a79-b827eb12c4d0/qt_motor-1.log].
log file: /home/qtrobot/.ros/log/4ee520d6-c950-11eb-9a79-b827eb12c4d0/qt_motor-1*.log
Initiating shutdown!
================================================================================
[robot_state_publisher-3] killing on exit
[controller-2] killing on exit
[INFO] [1623280196.395157]: Shutting down spawner. Stopping and unloading controllers...
[qt_motor-1] killing on exit
[INFO] [1623280196.396888]: Stopping all controllers...
[WARN] [1623280196.407480]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
shutting down processing monitor...
... shutting down processing monitor complete
done

This has only happened once.

Secondly when using the first option (trajectory controller) I can't see any controllers in the controller manager nor the joint trajectory controller: conttoller_manager joint_trajectory_controller Maybe something isn't starting as it should? Maybe I do something wrong?

Using the second option (skeleton detection) we have problems detecting the face when not the full body (or largee part of the body) is visible. Often users are standing or sitting too close to the robot for it to detect the skeleton and then it can't track the face. I have also been testing the built-in face detector (/qt_nuitrack_app/faces) but it only publish around 0.7 Hz which is not possible to use for face tracking.

Thanks again for your answers!

apaikan commented 3 years ago

Well, can you try running qt_motor_advanced.launch from robot autostart scripts and restart the robot? I imagine the issue is related to the unloaded requited parameters (rosparams) for qt_motor_advanced. This should work out-of-the shelve since we have used it many times. Anyway, @dkcodeguy (Denis) will try it again and let you know if there is any recent issue with that.

Regarding the nuitrack:

Cheers!

TechnoX commented 3 years ago

Well, can you try running qt_motor_advanced.launch from robot autostart scripts and restart the robot? I imagine the issue is related to the unloaded requited parameters (rosparams) for qt_motor_advanced. This should work out-of-the shelve since we have used it many times. Anyway, @dkcodeguy (Denis) will try it again and let you know if there is any recent issue with that.

Now I've tested both running roslaunch qt_motor qt_motor_advanced.launch from a terminal, and also changing ~/robot/autostart/start_qt_motor.sh and I can't see the controllers in the controller manager nor the joint trajectory controller.

However new topics are visible:

/qt_robot/head_controller/command
/qt_robot/head_controller/follow_joint_trajectory/cancel
/qt_robot/head_controller/follow_joint_trajectory/feedback
/qt_robot/head_controller/follow_joint_trajectory/goal
/qt_robot/head_controller/follow_joint_trajectory/result
/qt_robot/head_controller/follow_joint_trajectory/status
/qt_robot/head_controller/state
/qt_robot/joints/state
/qt_robot/left_arm_controller/command
/qt_robot/left_arm_controller/follow_joint_trajectory/cancel
/qt_robot/left_arm_controller/follow_joint_trajectory/feedback
/qt_robot/left_arm_controller/follow_joint_trajectory/goal
/qt_robot/left_arm_controller/follow_joint_trajectory/result
/qt_robot/left_arm_controller/follow_joint_trajectory/status
/qt_robot/left_arm_controller/state
/qt_robot/motors/states
/qt_robot/right_arm_controller/command
/qt_robot/right_arm_controller/follow_joint_trajectory/cancel
/qt_robot/right_arm_controller/follow_joint_trajectory/feedback
/qt_robot/right_arm_controller/follow_joint_trajectory/goal
/qt_robot/right_arm_controller/follow_joint_trajectory/result
/qt_robot/right_arm_controller/follow_joint_trajectory/status
/qt_robot/right_arm_controller/state

But I don't know if there is some problem since the controllers aren't visible in the RQT tools?

* Regarding the face output rate, just update the `<param name="face_frame_rate" value="2"/>` from [qt_nuitrack_app.launch](https://github.com/IntelRealSense/realsense-ros)  we tried even with 20Hrz

Great info! I will test this as well :)

apaikan commented 3 years ago

Hi @TechnoX . That's strange. From what you shared, I see all the trajectory controller topic are published correctly and the joint trajectory controller plugin of rqt must find the controller.

I have just repeated the scenario to ensure and here is the screenshot:

image

Just to clarify, did you chose the robot part (e.g. qt_robot/left_arm_conroller) for the controller list? (see image 2)

TechnoX commented 3 years ago

The controller dropdown list is not populated for me. So I can't select the robot part. As you can see in my first image the controller manager list is also empty: conttoller_manager

Can it be that the qt_motor node is running on the Raspberry and I'm opening the rqt_joint_trajectory_controller on the NUC (over ssh with the -X flag to forward the X-server) ? Since the raspberry is running Kinetic and the NUC is running Noetic the messages could have different versions?

I have had similar problems between the head and the nuc before, for example when I installed the ros-kinetic-audio-common-msgs in the head and ros-noetic-audio-common-msgs in the body I couldn't see the messages with rostopic echo, even though I saw the topics with rostopic list. So I decided to clone the audio_common_msgs repo in the catkin workspace instead and it worked right out of the box.

Are you running the rqt_joint_trajectory_controller from the NUC or the Raspberry? If you are running on the Raspberry, how can you do that without tunneling the X-server through the Nuc?

codionysus commented 3 years ago

Hi @TechnoX , I did some testing and for me running over ssh with the -X flag worked and also running directly from another ROS Noetic machine also. I didn't have issues with two different versions of ROS.

What helped me is to run motors in advance mode from the boot. Please modify "start_qt_motor.sh" on QTRP to run motors in advance mode (roslaunch qt_motor qt_motor_advanced.launch) from a reboot, because I think some messages or params don't set/unset when running from the command line. After you modified the file please reboot the QTrobot.

The first test that you can do is ssh with -X flag to QTRP like you did and run "rqt", select plugin rqt_joint_trajectory_controller, and check if you can control the motors. The second one you can directly run "rqt" from QTPC (not ssh to QTRP) and try commanding the motors from there.

Make sure that you first run just "rqt" by command line and then in rqt window select plugins and find "joint_trajectory_controller".

Hope this info helps you.

TechnoX commented 3 years ago

Okay, so I never got the controller to show up on the NUC. However if I connected to the Raspberry the dropdown list was populated with the controllers. So I decided to place my head_tracker_node on the raspberry. It works well! :)

I.e.

The first test that you can do is ssh with -X flag to QTRP like you did and run "rqt", select plugin rqt_joint_trajectory_controller, and check if you can control the motors.

Works fine! (But I can't ssh directly to the QTRP, I do first SSH to the NUC, and then to the QTRP. Both forwarding the X. Maybe you have some better way to directly ssh to the QTRP? I have never connected the QTRP to the wifi)

The second one you can directly run "rqt" from QTPC (not ssh to QTRP) and try commanding the motors from there.

This doesn't work.

And yes, I started the motor node in advanced mode at startup:

qtrobot@QTRD000185:~$ cat robot/autostart/start_qt_motor.sh 
# !/bin/bash

source /home/qtrobot/robot/autostart/qt_robot.inc

SCRIPT_NAME="start_qt_motor"
LOG_FILE=$(prepare_logfile "$SCRIPT_NAME")

{
prepare_ros_environment
wait_for_ros_node "/rosout" 60

roslaunch qt_motor qt_motor_advanced.launch

} &>> ${LOG_FILE}

I have written a new tracker according to Solution 1: Using trajectory controller (since it was the fastest to get up and running). But I still get some occasional crashes of the motor node (running in advanced mode).

qtrobot@QTRD000185:~$ cat robot/autostart/logs/start_qt_motor.log 

#############################################
 QT Robot autostart version <1.0>
 Running [start_qt_motor]
 Mon Jun 28 17:58:10 CEST 2021
#############################################

[Autostart] Preparing ROS environment [OK]
[Autostart] Waiting for ROS node /rosout
[Autostart] Waiting for TCPIP port 11311
[Autostart] TCPIP port 11311 [OK]
[Autostart] ROS node /rosout [ok]
... logging to /home/qtrobot/.ros/log/a4d714ee-d829-11eb-a9cf-b827eb12c4d0/roslaunch-QTRD000185-1812.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://192.168.100.1:46447/

SUMMARY
========

CLEAR PARAMETERS
 * /controller/
 * /qt_motor/

PARAMETERS
 * /herkulex_control/loop_frequency: 30
 * /herkulex_control/motors/HeadPitch/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/HeadPitch/callibration/offset: 1.63
 * /herkulex_control/motors/HeadPitch/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/HeadPitch/id: 10
 * /herkulex_control/motors/HeadPitch/index: 1
 * /herkulex_control/motors/HeadPitch/overload/period: 70
 * /herkulex_control/motors/HeadPitch/overload/threshold: 1000
 * /herkulex_control/motors/HeadPitch/part: head
 * /herkulex_control/motors/HeadPitch/pid/dead_zone: 0.0
 * /herkulex_control/motors/HeadPitch/pid/kd: 150
 * /herkulex_control/motors/HeadPitch/pid/ki: 0
 * /herkulex_control/motors/HeadPitch/pid/kp: 150
 * /herkulex_control/motors/HeadPitch/pid/sat_offset: 254
 * /herkulex_control/motors/HeadPitch/pid/sat_slope: 3000
 * /herkulex_control/motors/HeadPitch/pos/home: 0.0
 * /herkulex_control/motors/HeadPitch/pos/max: 21.63
 * /herkulex_control/motors/HeadPitch/pos/min: -13.37
 * /herkulex_control/motors/HeadPitch/pos/park: 23
 * /herkulex_control/motors/HeadPitch/velocity/default: 10
 * /herkulex_control/motors/HeadPitch/velocity/max: 30
 * /herkulex_control/motors/HeadYaw/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/HeadYaw/callibration/offset: -3.58
 * /herkulex_control/motors/HeadYaw/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/HeadYaw/id: 5
 * /herkulex_control/motors/HeadYaw/index: 0
 * /herkulex_control/motors/HeadYaw/overload/period: 20
 * /herkulex_control/motors/HeadYaw/overload/threshold: 400
 * /herkulex_control/motors/HeadYaw/part: head
 * /herkulex_control/motors/HeadYaw/pid/dead_zone: 0.0
 * /herkulex_control/motors/HeadYaw/pid/kd: 100
 * /herkulex_control/motors/HeadYaw/pid/ki: 0
 * /herkulex_control/motors/HeadYaw/pid/kp: 100
 * /herkulex_control/motors/HeadYaw/pid/sat_offset: 250
 * /herkulex_control/motors/HeadYaw/pid/sat_slope: 2000
 * /herkulex_control/motors/HeadYaw/pos/home: 0
 * /herkulex_control/motors/HeadYaw/pos/max: 54.42
 * /herkulex_control/motors/HeadYaw/pos/min: -61.58
 * /herkulex_control/motors/HeadYaw/pos/park: 0
 * /herkulex_control/motors/HeadYaw/velocity/default: 10
 * /herkulex_control/motors/HeadYaw/velocity/max: 100
 * /herkulex_control/motors/LeftElbowRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/LeftElbowRoll/callibration/offset: -2.27
 * /herkulex_control/motors/LeftElbowRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/LeftElbowRoll/id: 8
 * /herkulex_control/motors/LeftElbowRoll/index: 7
 * /herkulex_control/motors/LeftElbowRoll/overload/period: 20
 * /herkulex_control/motors/LeftElbowRoll/overload/threshold: 400
 * /herkulex_control/motors/LeftElbowRoll/part: left_arm
 * /herkulex_control/motors/LeftElbowRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/LeftElbowRoll/pid/kd: 150
 * /herkulex_control/motors/LeftElbowRoll/pid/ki: 0
 * /herkulex_control/motors/LeftElbowRoll/pid/kp: 150
 * /herkulex_control/motors/LeftElbowRoll/pid/sat_offset: 254
 * /herkulex_control/motors/LeftElbowRoll/pid/sat_slope: 3000
 * /herkulex_control/motors/LeftElbowRoll/pos/home: -35.0
 * /herkulex_control/motors/LeftElbowRoll/pos/max: -7.27
 * /herkulex_control/motors/LeftElbowRoll/pos/min: -82.27
 * /herkulex_control/motors/LeftElbowRoll/pos/park: -10.0
 * /herkulex_control/motors/LeftElbowRoll/velocity/default: 50
 * /herkulex_control/motors/LeftElbowRoll/velocity/max: 100
 * /herkulex_control/motors/LeftShoulderPitch/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/LeftShoulderPitch/callibration/offset: 2.6
 * /herkulex_control/motors/LeftShoulderPitch/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/LeftShoulderPitch/id: 6
 * /herkulex_control/motors/LeftShoulderPitch/index: 5
 * /herkulex_control/motors/LeftShoulderPitch/overload/period: 20
 * /herkulex_control/motors/LeftShoulderPitch/overload/threshold: 600
 * /herkulex_control/motors/LeftShoulderPitch/part: left_arm
 * /herkulex_control/motors/LeftShoulderPitch/pid/dead_zone: 0.0
 * /herkulex_control/motors/LeftShoulderPitch/pid/kd: 150
 * /herkulex_control/motors/LeftShoulderPitch/pid/ki: 0
 * /herkulex_control/motors/LeftShoulderPitch/pid/kp: 150
 * /herkulex_control/motors/LeftShoulderPitch/pid/sat_offset: 254
 * /herkulex_control/motors/LeftShoulderPitch/pid/sat_slope: 3000
 * /herkulex_control/motors/LeftShoulderPitch/pos/home: 90.0
 * /herkulex_control/motors/LeftShoulderPitch/pos/max: 140.6
 * /herkulex_control/motors/LeftShoulderPitch/pos/min: -135.4
 * /herkulex_control/motors/LeftShoulderPitch/pos/park: 90.0
 * /herkulex_control/motors/LeftShoulderPitch/velocity/default: 50
 * /herkulex_control/motors/LeftShoulderPitch/velocity/max: 100
 * /herkulex_control/motors/LeftShoulderRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/LeftShoulderRoll/callibration/offset: 1.63
 * /herkulex_control/motors/LeftShoulderRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/LeftShoulderRoll/id: 7
 * /herkulex_control/motors/LeftShoulderRoll/index: 6
 * /herkulex_control/motors/LeftShoulderRoll/overload/period: 20
 * /herkulex_control/motors/LeftShoulderRoll/overload/threshold: 600
 * /herkulex_control/motors/LeftShoulderRoll/part: left_arm
 * /herkulex_control/motors/LeftShoulderRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/LeftShoulderRoll/pid/kd: 150
 * /herkulex_control/motors/LeftShoulderRoll/pid/ki: 0
 * /herkulex_control/motors/LeftShoulderRoll/pid/kp: 150
 * /herkulex_control/motors/LeftShoulderRoll/pid/sat_offset: 254
 * /herkulex_control/motors/LeftShoulderRoll/pid/sat_slope: 3000
 * /herkulex_control/motors/LeftShoulderRoll/pos/home: -55.0
 * /herkulex_control/motors/LeftShoulderRoll/pos/max: -13.37
 * /herkulex_control/motors/LeftShoulderRoll/pos/min: -78.37
 * /herkulex_control/motors/LeftShoulderRoll/pos/park: -65.0
 * /herkulex_control/motors/LeftShoulderRoll/velocity/default: 50
 * /herkulex_control/motors/LeftShoulderRoll/velocity/max: 100
 * /herkulex_control/motors/RightElbowRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/RightElbowRoll/callibration/offset: -9.75
 * /herkulex_control/motors/RightElbowRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/RightElbowRoll/id: 2
 * /herkulex_control/motors/RightElbowRoll/index: 4
 * /herkulex_control/motors/RightElbowRoll/overload/period: 20
 * /herkulex_control/motors/RightElbowRoll/overload/threshold: 400
 * /herkulex_control/motors/RightElbowRoll/part: right_arm
 * /herkulex_control/motors/RightElbowRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/RightElbowRoll/pid/kd: 150
 * /herkulex_control/motors/RightElbowRoll/pid/ki: 0
 * /herkulex_control/motors/RightElbowRoll/pid/kp: 150
 * /herkulex_control/motors/RightElbowRoll/pid/sat_offset: 254
 * /herkulex_control/motors/RightElbowRoll/pid/sat_slope: 3000
 * /herkulex_control/motors/RightElbowRoll/pos/home: -35.0
 * /herkulex_control/motors/RightElbowRoll/pos/max: -14.75
 * /herkulex_control/motors/RightElbowRoll/pos/min: -89.75
 * /herkulex_control/motors/RightElbowRoll/pos/park: -10.0
 * /herkulex_control/motors/RightElbowRoll/velocity/default: 50
 * /herkulex_control/motors/RightElbowRoll/velocity/max: 100
 * /herkulex_control/motors/RightShoulderPitch/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/RightShoulderPitch/callibration/offset: -4.23
 * /herkulex_control/motors/RightShoulderPitch/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/RightShoulderPitch/id: 4
 * /herkulex_control/motors/RightShoulderPitch/index: 2
 * /herkulex_control/motors/RightShoulderPitch/overload/period: 20
 * /herkulex_control/motors/RightShoulderPitch/overload/threshold: 600
 * /herkulex_control/motors/RightShoulderPitch/part: right_arm
 * /herkulex_control/motors/RightShoulderPitch/pid/dead_zone: 0.0
 * /herkulex_control/motors/RightShoulderPitch/pid/kd: 150
 * /herkulex_control/motors/RightShoulderPitch/pid/ki: 0
 * /herkulex_control/motors/RightShoulderPitch/pid/kp: 150
 * /herkulex_control/motors/RightShoulderPitch/pid/sat_offset: 254
 * /herkulex_control/motors/RightShoulderPitch/pid/sat_slope: 3000
 * /herkulex_control/motors/RightShoulderPitch/pos/home: -90.0
 * /herkulex_control/motors/RightShoulderPitch/pos/max: 133.77
 * /herkulex_control/motors/RightShoulderPitch/pos/min: -142.23
 * /herkulex_control/motors/RightShoulderPitch/pos/park: -90.0
 * /herkulex_control/motors/RightShoulderPitch/velocity/default: 50
 * /herkulex_control/motors/RightShoulderPitch/velocity/max: 100
 * /herkulex_control/motors/RightShoulderRoll/callibration/inpos_margin: 0.5
 * /herkulex_control/motors/RightShoulderRoll/callibration/offset: 11.38
 * /herkulex_control/motors/RightShoulderRoll/callibration/stop_threshold: 0.5
 * /herkulex_control/motors/RightShoulderRoll/id: 3
 * /herkulex_control/motors/RightShoulderRoll/index: 3
 * /herkulex_control/motors/RightShoulderRoll/overload/period: 20
 * /herkulex_control/motors/RightShoulderRoll/overload/threshold: 600
 * /herkulex_control/motors/RightShoulderRoll/part: right_arm
 * /herkulex_control/motors/RightShoulderRoll/pid/dead_zone: 0.0
 * /herkulex_control/motors/RightShoulderRoll/pid/kd: 150
 * /herkulex_control/motors/RightShoulderRoll/pid/ki: 0
 * /herkulex_control/motors/RightShoulderRoll/pid/kp: 150
 * /herkulex_control/motors/RightShoulderRoll/pid/sat_offset: 254
 * /herkulex_control/motors/RightShoulderRoll/pid/sat_slope: 3000
 * /herkulex_control/motors/RightShoulderRoll/pos/home: -55.0
 * /herkulex_control/motors/RightShoulderRoll/pos/max: -3.62
 * /herkulex_control/motors/RightShoulderRoll/pos/min: -68.62
 * /herkulex_control/motors/RightShoulderRoll/pos/park: -65.0
 * /herkulex_control/motors/RightShoulderRoll/velocity/default: 50
 * /herkulex_control/motors/RightShoulderRoll/velocity/max: 100
 * /herkulex_control/serial_baudrate: 115200
 * /herkulex_control/serial_interface: /dev/serial0
 * /herkulex_control/serial_read_timeout: 10
 * /herkulex_control/status_frequency: 1
 * /herkulex_control/unit_radian: True
 * /qt_robot/HeadPitch_position/joint: HeadPitch
 * /qt_robot/HeadPitch_position/pid/d: 0.0
 * /qt_robot/HeadPitch_position/pid/i: 1.0
 * /qt_robot/HeadPitch_position/pid/p: 1.0
 * /qt_robot/HeadPitch_position/type: position_controll...
 * /qt_robot/HeadYaw_position/joint: HeadYaw
 * /qt_robot/HeadYaw_position/pid/d: 0.0
 * /qt_robot/HeadYaw_position/pid/i: 1.0
 * /qt_robot/HeadYaw_position/pid/p: 1.0
 * /qt_robot/HeadYaw_position/type: position_controll...
 * /qt_robot/LeftElbowRoll_position/joint: LeftElbowRoll
 * /qt_robot/LeftElbowRoll_position/pid/d: 0.0
 * /qt_robot/LeftElbowRoll_position/pid/i: 1.0
 * /qt_robot/LeftElbowRoll_position/pid/p: 1.0
 * /qt_robot/LeftElbowRoll_position/type: position_controll...
 * /qt_robot/LeftShoulderPitch_position/joint: LeftShoulderPitch
 * /qt_robot/LeftShoulderPitch_position/pid/d: 0.0
 * /qt_robot/LeftShoulderPitch_position/pid/i: 1.0
 * /qt_robot/LeftShoulderPitch_position/pid/p: 1.0
 * /qt_robot/LeftShoulderPitch_position/type: position_controll...
 * /qt_robot/LeftShoulderRoll_position/joint: LeftShoulderRoll
 * /qt_robot/LeftShoulderRoll_position/pid/d: 0.0
 * /qt_robot/LeftShoulderRoll_position/pid/i: 1.0
 * /qt_robot/LeftShoulderRoll_position/pid/p: 1.0
 * /qt_robot/LeftShoulderRoll_position/type: position_controll...
 * /qt_robot/RightElbowRoll_position/joint: RightElbowRoll
 * /qt_robot/RightElbowRoll_position/pid/d: 0.0
 * /qt_robot/RightElbowRoll_position/pid/i: 1.0
 * /qt_robot/RightElbowRoll_position/pid/p: 1.0
 * /qt_robot/RightElbowRoll_position/type: position_controll...
 * /qt_robot/RightShoulderPitch_position/joint: RightShoulderPitch
 * /qt_robot/RightShoulderPitch_position/pid/d: 0.0
 * /qt_robot/RightShoulderPitch_position/pid/i: 1.0
 * /qt_robot/RightShoulderPitch_position/pid/p: 1.0
 * /qt_robot/RightShoulderPitch_position/type: position_controll...
 * /qt_robot/RightShoulderRoll_position/joint: RightShoulderRoll
 * /qt_robot/RightShoulderRoll_position/pid/d: 0.0
 * /qt_robot/RightShoulderRoll_position/pid/i: 1.0
 * /qt_robot/RightShoulderRoll_position/pid/p: 1.0
 * /qt_robot/RightShoulderRoll_position/type: position_controll...
 * /qt_robot/gesture/adjust_mainloop_frequency: False
 * /qt_robot/gesture/gestures_path: /home/qtrobot/rob...
 * /qt_robot/gesture/type: qt_gesture_contro...
 * /qt_robot/head_controller/joints: ['HeadYaw', 'Head...
 * /qt_robot/head_controller/type: position_controll...
 * /qt_robot/head_position/joints: ['HeadYaw', 'Head...
 * /qt_robot/head_position/type: position_controll...
 * /qt_robot/joints_state/publish_rate: 30
 * /qt_robot/joints_state/type: joint_state_contr...
 * /qt_robot/left_arm_controller/joints: ['LeftShoulderPit...
 * /qt_robot/left_arm_controller/type: position_controll...
 * /qt_robot/left_arm_position/joints: ['LeftShoulderPit...
 * /qt_robot/left_arm_position/type: position_controll...
 * /qt_robot/motors/motors_state_publish_rate: 1
 * /qt_robot/motors/type: qt_motors_control...
 * /qt_robot/right_arm_controller/joints: ['RightShoulderPi...
 * /qt_robot/right_arm_controller/type: position_controll...
 * /qt_robot/right_arm_position/joints: ['RightShoulderPi...
 * /qt_robot/right_arm_position/type: position_controll...
 * /robot_description: <?xml version="1....
 * /robot_description_kinematics/left_arm/kinematics_solver: qtrobot_left_arm_...
 * /robot_description_kinematics/left_arm/kinematics_solver_attempts: 3
 * /robot_description_kinematics/left_arm/kinematics_solver_search_resolution: 0.005
 * /robot_description_kinematics/left_arm/kinematics_solver_timeout: 0.005
 * /robot_description_kinematics/right_arm/kinematics_solver: qtrobot_right_arm...
 * /robot_description_kinematics/right_arm/kinematics_solver_attempts: 3
 * /robot_description_kinematics/right_arm/kinematics_solver_search_resolution: 0.005
 * /robot_description_kinematics/right_arm/kinematics_solver_timeout: 0.005
 * /robot_description_planning/joint_limits/HeadPitch/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/HeadPitch/has_velocity_limits: True
 * /robot_description_planning/joint_limits/HeadPitch/max_acceleration: 0
 * /robot_description_planning/joint_limits/HeadPitch/max_velocity: 30
 * /robot_description_planning/joint_limits/HeadYaw/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/HeadYaw/has_velocity_limits: True
 * /robot_description_planning/joint_limits/HeadYaw/max_acceleration: 0
 * /robot_description_planning/joint_limits/HeadYaw/max_velocity: 100
 * /robot_description_planning/joint_limits/LeftElbowRoll/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/LeftElbowRoll/has_velocity_limits: True
 * /robot_description_planning/joint_limits/LeftElbowRoll/max_acceleration: 0
 * /robot_description_planning/joint_limits/LeftElbowRoll/max_velocity: 100
 * /robot_description_planning/joint_limits/LeftShoulderPitch/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/LeftShoulderPitch/has_velocity_limits: True
 * /robot_description_planning/joint_limits/LeftShoulderPitch/max_acceleration: 0
 * /robot_description_planning/joint_limits/LeftShoulderPitch/max_velocity: 100
 * /robot_description_planning/joint_limits/LeftShoulderRoll/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/LeftShoulderRoll/has_velocity_limits: True
 * /robot_description_planning/joint_limits/LeftShoulderRoll/max_acceleration: 0
 * /robot_description_planning/joint_limits/LeftShoulderRoll/max_velocity: 100
 * /robot_description_planning/joint_limits/RightElbowRoll/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/RightElbowRoll/has_velocity_limits: True
 * /robot_description_planning/joint_limits/RightElbowRoll/max_acceleration: 0
 * /robot_description_planning/joint_limits/RightElbowRoll/max_velocity: 100
 * /robot_description_planning/joint_limits/RightShoulderPitch/has_acceleration_limits: False
 * /robot_description_planning/joint_li[ INFO] [1624895909.383606178]:   Found HeadPitch index:1
[ INFO] [1624895909.521020709]:      Found HeadYaw index:0
[ INFO] [1624895909.604961699]:      Found LeftElbowRoll index:7
[ INFO] [1624895909.673767011]:      Found LeftShoulderPitch index:5
[ INFO] [1624895909.729975761]:      Found LeftShoulderRoll index:6
[ INFO] [1624895909.795030866]:      Found RightElbowRoll index:4
[ INFO] [1624895909.858158834]:      Found RightShoulderPitch index:2
[ INFO] [1624895909.927527949]:      Found RightShoulderRoll index:3
[ INFO] [1624895909.997131490]: Loaded 8 motors configuration parameters!
[ INFO] [1624895910.015203209]: Opening herkulex motors driver... (baudrate: 115200)
[ INFO] [1624895910.516056022]: Rebooting motors...
[ INFO] [1624895913.543330968]: *** Setting temp/voltage limits ***
[ INFO] [1624895913.548947010]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (HeadYaw)
[ INFO] [1624895913.551983989]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (HeadPitch)
[ INFO] [1624895913.554978364]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (RightShoulderPitch)
[ INFO] [1624895913.557999302]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (RightShoulderRoll)
[ INFO] [1624895913.560929510]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (RightElbowRoll)
[ INFO] [1624895913.563947791]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (LeftShoulderPitch)
[ INFO] [1624895913.567004354]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (LeftShoulderRoll)
[ INFO] [1624895913.570039927]: Voltage: [6.44V - 11.47V], Temp: 84.11C  (LeftElbowRoll)
[ INFO] [1624895913.570162479]: ***********************************
[ INFO] [1624895913.570534510]: *** Setting callibration params ***
[ INFO] [1624895913.571028104]: *** Setting PID and compliancy ***
[ INFO] [1624895913.585520499]: *** Setting overload detection values ***
[ INFO] [1624895913.605779093]: *** Turning all motors torque ***
[ INFO] [1624895916.622809509]: *** Clearing all motor status ***
[ INFO] [1624895916.623497686]: *** Setting joint limits ***
[ INFO] [1624895916.624031853]: *** Reading joints current postion ***
[ INFO] [1624895916.638229405]: Pos motor HeadYaw : 0.30
[ INFO] [1624895916.641447113]: Pos motor HeadPitch : 0.60
[ INFO] [1624895916.644737061]: Pos motor RightShoulderPitch : -89.90
[ INFO] [1624895916.647854873]: Pos motor RightShoulderRoll : -57.30
[ INFO] [1624895916.650841540]: Pos motor RightElbowRoll : -35.20
[ INFO] [1624895916.653772894]: Pos motor LeftShoulderPitch : 89.90
[ INFO] [1624895916.656877998]: Pos motor LeftShoulderRoll : -57.00
[ INFO] [1624895916.659889977]: Pos motor LeftElbowRoll : -34.80
[ INFO] [1624895916.660800707]: *** Staring controller main loop ***
[ WARN] [1624895920.443508570]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624895920.444630236]: Cannot read behavior of motor HeadPitch!
[ WARN] [1624895921.465121017]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624895921.465275184]: Cannot read behavior of motor LeftShoulderPitch!
[ INFO] [1624895926.663326276]: QTMotorsController: starting
[ INFO] [1624895926.663520338]: QTGestureController: starting
[ WARN] [1624895944.728262708]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624895944.728504270]: Cannot read behavior of motor RightShoulderPitch!
[ WARN] [1624895947.555103696]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624895947.555721509]: Cannot read behavior of motor LeftShoulderPitch!
[ WARN] [1624896027.071536447]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896027.071691339]: Cannot read behavior of motor HeadPitch!
[ WARN] [1624896027.405123155]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896027.405424033]: Cannot read behavior of mototerminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error> >'
  what():  boost::thread_resource_error: Resource temporarily unavailable
r RightShoulderPitch!
[ WARN] [1624896029.545108727]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896029.545274817]: Cannot read behavior of motor RightElbowRoll!
[ WARN] [1624896030.052113989]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896030.052378203]: Cannot read behavior of motor LeftShoulderPitch!
[ WARN] [1624896033.418307158]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896033.418458926]: Cannot read behavior of motor RightShoulderPitch!
[ WARN] [1624896033.618239962]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896033.618410636]: Cannot read behavior of motor RightShoulderRoll!
[ WARN] [1624896794.365100487]: getMotorBehaviourRaw : servo is not responding
[ WARN] [1624896794.365240013]: Cannot read behavior of motor HeadYaw!
================================================================================REQUIRED process [qt_motor-1] has died!
process has died [pid 2020, exit code -6, cmd /opt/ros/kinetic/lib/qt_motor/qt_motor joint_states:=/qt_robot/joints/state __name:=qt_motor __log:=/home/qtrobot/.ros/log/a4d714ee-d829-11eb-a9cf-b827eb12c4d0/qt_motor-1.log].
log file: /home/qtrobot/.ros/log/a4d714ee-d829-11eb-a9cf-b827eb12c4d0/qt_motor-1*.log
Initiating shutdown!
================================================================================
[WARN] [1624896847.681525]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[INFO] [1624895925.502775]: Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [1624895925.512268]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1624895925.520295]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1624895925.528649]: Loading controller: /qt_robot/head_controller
[INFO] [1624895925.923557]: Loading controller: /qt_robot/right_arm_controller
[INFO] [1624895926.171046]: Loading controller: /qt_robot/left_arm_controller
[INFO] [1624895926.373040]: Loading controller: /qt_robot/joints_state
[INFO] [1624895926.421393]: Loading controller: /qt_robot/motors
[INFO] [1624895926.489461]: Loading controller: /qt_robot/gesture
[INFO] [1624895926.624164]: Controller Spawner: Loaded controllers: /qt_robot/head_controller, /qt_robot/right_arm_controller, /qt_robot/left_arm_controller, /qt_robot/joints_state, /qt_robot/motors, /qt_robot/gesture
[INFO] [1624895926.664741]: Started controllers: /qt_robot/head_controller, /qt_robot/right_arm_controller, /qt_robot/left_arm_controller, /qt_robot/joints_state, /qt_robot/motors, /qt_robot/gesture
[INFO] [1624896847.667186]: Shutting down spawner. Stopping and unloading controllers...
[INFO] [1624896847.669166]: Stopping all controllers...
mits/RightShoulderPitch/has_velocity_limits: True
 * /robot_description_planning/joint_limits/RightShoulderPitch/max_acceleration: 0
 * /robot_description_planning/joint_limits/RightShoulderPitch/max_velocity: 100
 * /robot_description_planning/joint_limits/RightShoulderRoll/has_acceleration_limits: False
 * /robot_description_planning/joint_limits/RightShoulderRoll/has_velocity_limits: True
 * /robot_description_planning/joint_limits/RightShoulderRoll/max_acceleration: 0
 * /robot_description_planning/joint_limits/RightShoulderRoll/max_velocity: 100
 * /robot_description_semantic: <?xml version="1....
 * /rosdistro: kinetic
 * /rosversion: 1.12.17

NODES
  /
    controller (controller_manager/spawner)
    qt_motor (qt_motor/qt_motor)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)

ROS_MASTER_URI=http://192.168.100.1:11311

process[qt_motor-1]: started with pid [2020]
process[controller-2]: started with pid [2021]
process[robot_state_publisher-3]: started with pid [2023]
[robot_state_publisher-3] killing on exit
[qt_motor-1] killing on exit
[controller-2] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

To create the error above, I'm publishing things to /qt_robot/head_controller/command and to /qt_robot/gesture/play at the same time. The error occured after approx 30 minutes.

apaikan commented 3 years ago

Thanks @TechnoX for detailed report. Regarding ssh to QTTP it's strange that you cannon ssh to it directly. Usually if your laptop connected to QT wifi, you should be able ssh to it using developer user from your laptop: ssh developer@192.168.100.1. Is there any issue with password?

Regarding motor controller crash report: thanks again for the the example. we will replicate it and do more in depth debugging.

TechnoX commented 3 years ago

Regarding ssh: Ah, I'm not connected to the QT wifi, I connected the QTPC to my own router to get Internet access. Or can it be connected to Internet at the same time?

Regarding motors: It would be very helpful if you can do some debugging during this week. Our client really wants the robot to be gesticulating and tracking faces simultaneous.

codionysus commented 3 years ago

Hi @TechnoX,

Regarding ssh : QTrobot has two Wifis, one of them is the access point (QTRP) and the second one is available for you to use it to connect to the internet. Please check the following image, also you can check on our documentation network configuration of QTrobot.

image

As explained in image above, if you connect QTPC to the internet, your QTRP should have also access to the internet and every device that connects to the QTrobot access point (QTRP) will also have internet. You can also check how to shh to QTrobot.

Regarding motors: I have tried to debug/replicate your issue:

  1. running qt_motors in advanced_mode
  2. script_1 sending commands to "/qt_robot/head_controller/command"

For the ('/qt_robot/gesture/play') I tried two things. Frist one is publishing to the topic and the other one is calling a service. When I was publishing to the topic, it crashed after approx. 30 minutes as you said, but when I tried with calling ('/qt_robot/gesture/play') service it didn't crash (it was running for 2h+). Check API of gesture interface.

We are still debugging it, but can you try calling the service instead of publishing to the topic? Let me know if this helps.

If you want, you can try running this code (uncomment topic or service part which you want to test):

#!/usr/bin/env python
import sys
import rospy
from std_msgs.msg import String
from qt_gesture_controller.srv import *

rospy.init_node('qt_motor_testing')

# Create Service Call
# gesturePlay = rospy.ServiceProxy('/qt_robot/gesture/play', gesture_play)

# create a publisher
gesturePlay = rospy.Publisher('/qt_robot/gesture/play', String, queue_size=10)

def play_gesture():
    try:
        #service call
        #gesturePlay("<name of your gesture>", 1.0)

        #topic
        gesturePlay.publish("<name of your gesture>")
    except rospy.ROSInterruptException:
        rospy.logerr("could not play the gesture!")

if __name__ == '__main__':
    # wait for publisher/subscriber connections
    wtime_begin = rospy.get_time()
    while (gesturePlay.get_num_connections() == 0) :
        rospy.loginfo("waiting for subscriber connections...")
        if rospy.get_time() - wtime_begin > 10.0:
            rospy.logerr("Timeout while waiting for subscribers connection!")
            sys.exit()
        rospy.sleep(1)

    while True:
        play_gesture()
        rospy.loginfo("played gesture...")
        rospy.sleep(1)
        if rospy.is_shutdown():
                    break
codionysus commented 3 years ago

Hi @TechnoX,

There was threading issue and we fixed it, thanks for letting us know. We have update qt_gesture_controller package with the new fixes.

SSH to QTRP and download the new qt_gesture_controller with this command :

wget https://bitbucket.org/qtrobot/packages/raw/f0d85b4a33a8b2404deae1f7adecc37cad288719/deb/ros-kinetic-qt-gesture-controller_1.4.0-0xenial_armhf.deb

Install the package:

sudo dpkg -i ros-kinetic-qt-gesture-controller_1.4.0-0xenial_armhf.deb

Then just reboot QTrobot or relaunch qt_motor interface. You can call service or publish to the topic ("/qt_robot/gesture/play"), both were tested and qt_motor interface didn't crash.

TechnoX commented 2 years ago

Just wanted to say that the threading issue seems to be fixed since we have been using the robot during the summer without any problems. Thanks! So I will close this issue.

However I've seen another error, but create a separate issue #52 for that.