Open Kanazawanaoaki opened 2 years ago
先程もpr1040でこの問題と同じようにサーボが入らない事態が発生しました.一時的な解決策としては,やはり(本当に効果があるかは未検証だが)pr2の腕を動かすなどして暫くするとサーボが入るようになるようです.
まだ原因や解決策を解明できた訳ではないので,分かっている情報をなるべく整理して記しておきます.
ログなど:https://gist.github.com/Kanazawanaoaki/206e1ef6452711b7d648b1e7f556bac5
サーボが入らなくなる時の症状としては,
audible warning
で realtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.;
が出ている.(hzのwarningだけの時はサーボが入る)robot.log
に [ERROR] [1664324590.399660822] [/realtime_loop:ros.ethercat_hardware]: Device 16 : previous port error counters less current values
のようなエラーが出ているrx error
が発生しているの3つの症状が発生しているようです.それぞれの関係性と原因についてはまだ良くわかっていません.
audible warning
で realtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.;
が出ている.2022年の9月に入ってからpr1040でRealtime loopが遅くなり audible warning
で以下のようなwarningが出ることが報告されるようになった.https://github.com/jsk-ros-pkg/jsk_robot/issues/1623
[INFO] [1664417978.378555] [/audible_warning:rosout]: audible warning talking: error. realtime controllers realtime control loop halting, realtime loop only ran at 740.0000 hz
さらにrealtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.;
が同時に表示されていることがある.
[INFO] [1665932713.034740] [/audible_warning:rosout]: audible warning talking: error. realtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.; halting, realtime loop only ran at 740.0000 hz
このようなwarningの場合はサーボを入れるとwarningが出なくなる(しばらくすると再発することもある)ことが報告されています https://github.com/jsk-ros-pkg/jsk_robot/issues/1623#issuecomment-1267112863 . 確かに昨日のログでも今日のログでもランストップを押してサーボを入れるとそれまで連続していたwarningが出なくなっている.
これは,https://github.com/jsk-ros-pkg/jsk_robot/issues/1623#issuecomment-1267150609 で言われているように
ということのようである.
しかし,以下のようにrealtime loop used too much time in the last 30 second
のwarningも出ている時にランストップを入れるとサーボが入らないようである.(昨日の該当箇所のログ,今日の該当箇所のログ)
[INFO] [1664309197.368809] [/audible_warning:rosout]: audible warning talking: error. realtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.; halting, realtime loop only ran at 740.0000 hz
これまでもhzのwarningとtoo muchのwarningが一緒に出ていたことはあるようだが,そのタイミングでサーボを入れていなかったからこの問題が発生しなかっただけ?
この情報で検索してhitしたのは以下の2つ. 詳しくみる必要があるかも.
robot.log
に [ERROR] [1664324590.399660822] [/realtime_loop:ros.ethercat_hardware]: Device 16 : previous port error counters less current values
のようなエラーが出ているよく調べていくと昨日のログには
readMailboxInternal : [43mERROR[0m asking for read repeat after dropping 1 packets
だけではなく,以下のエラーログも出ていた.今日のログにもDevice の番号は違うが出ていたので,これがサーボがつかない直接的な原因かもしれない.
[31m[ERROR] [1665826174.272734993] [/realtime_loop:ros.ethercat_hardware]: Device 16 : previous port error counters less current values[0m
このエラーログはPR2/pr2_ethercat_driversで出されているものだということは分かった.
https://github.com/PR2/pr2_ethercat_drivers/blob/d93eb53a0a3a8e6706f7061e30e37a5d021eee1d/ethercat_hardware/src/ethercat_device.cpp#L245-L247
rx error
が発生しているhttps://github.com/jsk-ros-pkg/jsk_robot/issues/1671#issue-1410142269 でも報告した通り,rx errorが発生している時にサーボが入らなくなり,サーボが安定している時にはrx errorは増えていない.これが根本の原因の可能性もある.
$ rosrun jsk_pr2_startup diagnostics_rx_error_summarizer.py
/Motors/EtherCAT Device #00 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 208261
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 0: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 1: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Master RX Errors: 288952
[Max Device]
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 208261
Forwarded 以外の RX Error Port でみると,やはり l_elbow_flex_motor
でエラーがおきているように思える.サーボが復活した時にはMasterもl_elbow_flex_motorもrx errorが上がらなくなった.
/Motors/EtherCAT Master RX Errors
が何を示しているか等は不明.RX Error Portの合計値という訳でも無い.しかしethtool -S ecat0
したときのrx_crc_errors
とrx_frame_errors
の値の合計がrx_errors
になっている.
(rx_errors) = (rx_crc_errors) + (rx_frame_errors)
ちなみに関連していそうな話題としてEtherCAT Master
においてros wiki に載っている以下の関係式があるが RX Errorとの関連性は不明.値は微妙に一致していない...
(Dropped Packets) = (CRC Errors) + (RX Late Packet)
値の一覧
rx_errors: 460752
rx_crc_errors: 230412
rx_frame_errors: 230340
Dropped Packets: 154664
RX Late Packet: 8
CRC Errors (Dropped Packets - RX Late Packet): 154656
/Motors/EtherCAT Device #10 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #10 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #10 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #10 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #11 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #11 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #11 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #11 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #35 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #35 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #35 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #35 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #36 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #36 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #36 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #36 (WG014) RX Error Port 3: 44
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 0: 255
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 8
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 0: 83767
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 0: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 1: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 0: 180355
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 1: 2
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 1: 24048
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 1: 1033
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Master RX Errors: 347906
[Max Device]
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 0: 180355
https://gist.github.com/Kanazawanaoaki/d8d60a1b308f9575b769d0c46e02c900
https://drive.google.com/drive/folders/1T0TSiWjo1XM5nHTngxdGk-HBJfa9dzx5?usp=sharing
realtime loop used too much time in the last 30 seconds.;
が発生していたlogは
jsk-pr2-startup.log.12
jsk-pr2-startup.log.10
jsk-pr2-startup.log.9
jsk-pr2-startup.log.5
Device 27 : previous port error counters less current values
のエラーが発生していたlogは
robot.log.10
robot.log.9
robot.log.5
robot.log.1
完全に一致してはいないが関係していそう.
https://gist.github.com/Kanazawanaoaki/bc6d7080c3f04ab8428b36979d853abe
https://drive.google.com/drive/folders/1B7cn8lGbUyinELv_V-ctNEr39lTaacNc?usp=sharing
realtime loop used too much time in the last 30 seconds.;
が発生していたlogは
jsk-pr2-startup.log.14
jsk-pr2-startup.log.13
jsk-pr2-startup.log.12
jsk-pr2-startup.log.9
jsk-pr2-startup.log.8
jsk-pr2-startup.log.7
jsk-pr2-startup.log.3
jsk-pr2-startup.log.2
jsk-pr2-startup.log.1
jsk-pr2-startup.log
Device 27 : previous port error counters less current values
のエラーが発生していたlogは
robot.log.14
robot.log.13
robot.log.9
robot.log.8
robot.log.7
robot.log.1
robot.log
関連はしているが,必ず同時に起こる症状ではないのかもしれない.
こちらの問題ですが,l_elbow_flex_motor
とl_forearm_roll_motor
の間の通信経路やスリップリングの接触不良があるような気がしています.
/Motors/EtherCAT Device #00 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 92621
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 0: 362
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 0: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 1: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Master RX Errors: 270586
[Max Device]
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 92621
--------------------------------------------
実際に,今調べてみると,l_elbow_flex_motor
とl_forearm_roll_motor
でRX Errorが発生していることを確認しました.今後の予定ですが,本格的に直すにはここの接続をバイパスしてチェックしてみるとかが必要になるかもしれません.
cc. @knorth55
先程も同じようなエラーが発生しました.
audible warning
で realtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.;
が出ている.[INFO] [1666323521.378856] [/audible_warning:rosout]: audible warning talking: error. realtime controllers realtime control loop realtime loop used too much time in the last 30 seconds.; halting, realtime loop only ran at 735.0000 hz
というaudible_warningを喋っていたので,確認の為にRun Stopをonにした所,やはりサーボが上手くつかずガクガクする状態になりました.
robot.log
に [ERROR] [1664324590.399660822] [/realtime_loop:ros.ethercat_hardware]: Device 16 : previous port error counters less current values
のようなエラーが出ている~robot.logを確認すると,
readMailboxInternal : ERROR asking for read repeat after dropping 1 packets
のログは出ていましたが,https://github.com/jsk-ros-pkg/jsk_robot/issues/1671#issuecomment-1280037795 で報告していた[/realtime_loop:ros.ethercat_hardware]: Device 16 : previous port error counters less current values
のエラーは出ていませんでした.このエラーは本件とは直接は関係しないエラーのかも知れません.
rx error
が発生しているrx_errorを確認すると今度は,l_forearm_roll_motor
でのみ発生していて,サーボが落ちている(or ガクガクしている)時には値が増えていきました.
/Motors/EtherCAT Device #00 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #00 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #01 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #15 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 0: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 1: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 2: 0
/Motors/EtherCAT Device #16 (WG014) RX Error Port 3: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (bl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (br_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fl_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_l_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_r_wheel_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (fr_caster_rotation_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (head_tilt_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 0: 9120
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (l_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (laser_tilt_mount_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 0: 0
/Motors/EtherCAT Device (led_projector) RX Error Port 1: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_elbow_flex_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_forearm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_gripper_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_shoulder_pan_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_upper_arm_roll_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_l_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (r_wrist_r_motor) RX Error Port 1: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 0: 0
/Motors/EtherCAT Device (torso_lift_motor) RX Error Port 1: 0
/Motors/EtherCAT Master RX Errors: 10295
[Devices with errors]
/Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 0: 9120
--------------------------------------------
各デバイスでのDropsも確認しましたがやはり0でした.
EtherCAT Master Dropped Packets: 5578
EtherCAT Master RX Late Packet: 0
EtherCAT Master CRC Errors (Dropped Packets - RX Late Packet): 5578
EtherCAT Device (br_caster_r_wheel_motor) Drops: 0
EtherCAT Device (br_caster_l_wheel_motor) Drops: 0
EtherCAT Device (br_caster_rotation_motor) Drops: 0
EtherCAT Device (fl_caster_r_wheel_motor) Drops: 0
EtherCAT Device (fl_caster_l_wheel_motor) Drops: 0
EtherCAT Device (fl_caster_rotation_motor) Drops: 0
EtherCAT Device (bl_caster_r_wheel_motor) Drops: 0
EtherCAT Device (bl_caster_l_wheel_motor) Drops: 0
EtherCAT Device (bl_caster_rotation_motor) Drops: 0
EtherCAT Device (fr_caster_r_wheel_motor) Drops: 0
EtherCAT Device (fr_caster_l_wheel_motor) Drops: 0
EtherCAT Device (fr_caster_rotation_motor) Drops: 0
EtherCAT Device (torso_lift_motor) Drops: 0
EtherCAT Device (l_shoulder_pan_motor) Drops: 0
EtherCAT Device (l_shoulder_lift_motor) Drops: 0
EtherCAT Device (l_upper_arm_roll_motor) Drops: 0
EtherCAT Device (l_elbow_flex_motor) Drops: 0
EtherCAT Device (l_forearm_roll_motor) Drops: 0
EtherCAT Device (l_wrist_l_motor) Drops: 0
EtherCAT Device (l_wrist_r_motor) Drops: 0
EtherCAT Device (l_gripper_motor) Drops: 0
EtherCAT Device (head_pan_motor) Drops: 0
EtherCAT Device (r_shoulder_pan_motor) Drops: 0
EtherCAT Device (r_shoulder_lift_motor) Drops: 0
EtherCAT Device (r_upper_arm_roll_motor) Drops: 0
EtherCAT Device (r_elbow_flex_motor) Drops: 0
EtherCAT Device (r_forearm_roll_motor) Drops: 0
EtherCAT Device (r_wrist_l_motor) Drops: 0
EtherCAT Device (r_wrist_r_motor) Drops: 0
EtherCAT Device (r_gripper_motor) Drops: 0
EtherCAT Device (head_tilt_motor) Drops: 0
EtherCAT Device (laser_tilt_mount_motor) Drops: 0
--------------------------------------------
l_elbow_flex_motor
とl_forearm_roll_motor
の間の通信経路やスリップリングの接触不良がある可能性がさらに高くなりました.暫くするとrx_errorも発生せず,サーボが安定して入るようになりました.
同じような問題が現在も発生していて, https://github.com/jsk-ros-pkg/jsk_robot/issues/794 と同じように,
clearReadMailbox : WARN read mbx contained garbage data
が出ていてサーボが入らないことがある.
--------------------------------------------
EtherCAT Master Dropped Packets: 4893961
EtherCAT Master RX Late Packet: 0
EtherCAT Device (br_caster_r_wheel_motor)Drops: 0
EtherCAT Device (br_caster_l_wheel_motor)Drops: 0
EtherCAT Device (br_caster_rotation_motor)Drops: 0
EtherCAT Device (fl_caster_r_wheel_motor)Drops: 0
EtherCAT Device (fl_caster_l_wheel_motor)Drops: 0
EtherCAT Device (fl_caster_rotation_motor)Drops: 0
EtherCAT Device (bl_caster_r_wheel_motor)Drops: 0
EtherCAT Device (bl_caster_l_wheel_motor)Drops: 0
EtherCAT Device (bl_caster_rotation_motor)Drops: 0
EtherCAT Device (fr_caster_r_wheel_motor)Drops: 0
EtherCAT Device (fr_caster_l_wheel_motor)Drops: 0
EtherCAT Device (fr_caster_rotation_motor)Drops: 0
EtherCAT Device (torso_lift_motor)Drops: 0
EtherCAT Device (l_shoulder_pan_motor)Drops: 0
EtherCAT Device (l_shoulder_lift_motor)Drops: 0
EtherCAT Device (l_upper_arm_roll_motor)Drops: 0
EtherCAT Device (l_elbow_flex_motor)Drops: 0
EtherCAT Device (l_forearm_roll_motor)Drops: 10153
EtherCAT Device (l_wrist_l_motor)Drops: 10153
EtherCAT Device (l_wrist_r_motor)Drops: 10153
EtherCAT Device (l_gripper_motor)Drops: 10153
EtherCAT Device (head_pan_motor)Drops: 0
EtherCAT Device (r_shoulder_pan_motor)Drops: 0
EtherCAT Device (r_shoulder_lift_motor)Drops: 0
EtherCAT Device (r_upper_arm_roll_motor)Drops: 0
EtherCAT Device (r_elbow_flex_motor)Drops: 0
EtherCAT Device (r_forearm_roll_motor)Drops: 0
EtherCAT Device (r_wrist_l_motor)Drops: 0
EtherCAT Device (r_wrist_r_motor)Drops: 0
EtherCAT Device (r_gripper_motor)Drops: 0
EtherCAT Device (head_tilt_motor)Drops: 0
EtherCAT Device (laser_tilt_mount_motor)Drops: 0
--------------------------------------------
[Devices with errors]
1696574170 : /Motors/EtherCAT Device (fl_caster_r_wheel_motor) RX Error Port 1: 2
1696574170 : /Motors/EtherCAT Device (l_elbow_flex_motor) RX Error Port 1: 167859
1696574170 : /Motors/EtherCAT Device (l_forearm_roll_motor) RX Error Port 0: 704
となっていてやはり左腕のl_elbow_flex_motor
とl_forearm_roll_motor
の辺りが怪しそう.
l_elbow_flex_motorとl_forearm_roll_motorの間の通信経路やスリップリングの接触不良がある可能性がさらに高くなりました.
1年前にはケーブルとかスリップリングは交換したり、接点復活剤を掛けたり、スリップリングを飛ばして直接ボード同士をケーブルでつないだりしたのかな?Noだったらまずはそれを試して、Yesだったら新しく着たアームの基板と交換してみるのかな。
一年前にも同じような状態で緊急ではないということで何もしていないのでNoで, スリップリングの換えは機械加工すれば使えるものがあるので交換するか,スリップリングを飛ばして直接ボード同士をケーブルを時期をみてやって様子をみることになるかと思います.
その2つだと
スリップリングを飛ばして直接ボード同士をケーブルを
◉ Kei Okada
2023年10月6日(金) 15:53 Naoaki Kanazawa @.***>:
一年前にも同じような状態で緊急ではないということで何もしていないのでNoで,
スリップリングの換えは機械加工すれば使えるものがあるので交換するか,スリップリングを飛ばして直接ボード同士をケーブルを時期をみてやって様子をみることになるかと思います.
— Reply to this email directly, view it on GitHub https://github.com/jsk-ros-pkg/jsk_robot/issues/1671#issuecomment-1750075212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYNXDFH6JCMWGJXDNPC4LX56THJAVCNFSM6AAAAAARF4EVISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQGA3TKMRRGI . You are receiving this because you commented.Message ID: @.***>
pr1040でサーボが入らなくなる(入ってもガクガクする)問題が発生し,原因を調べた所,https://github.com/jsk-ros-pkg/jsk_robot/issues/794 と同じようにrx errorが多くなっていました.
原因特定の経緯
pr1040が暫くaudible warningでrealtime loopについて喋っていたのでサーボを入れてみたが入らなかった.(realtime loopのwarningは一度サーボを入れると無くなるという情報有り?要出典調査)
のログをみると2022/10/15 14:36:54 あたりから
のようなwarningを喋っていたよう.
rqt_pr2_dashboardを見ると,モーターではパケットドロップしていないがMasterではドロップしている. (時々 “Could not collect WG0X diagnostics” のようなメッセージも出ている)
robotのログを見ると
が出ている.これを検索すると以下のpr1012でのEtherCATのNICでrx errorが多いというissueがhit. https://github.com/jsk-ros-pkg/jsk_robot/issues/794 issueを見るとこれ自体はサーボが入らなくなる様なエラーでは無い気がするが,issueで出てきている
ethtool -S ecat0
を試してみるとrx_errorsが多かった.※ログが流れてしまったので,後ほど20:01時点で実行した結果,
issueで報告されているpr1040の値は rx_errors: 4 なのでとても多くなっている.
rx_errorsをcountするscriptで確認.出力されたcsvファイルを見ると,l_elbow_flex_motorのRX Error Port 1の値が大きく,上昇もしているので怪しい.
rqt_pr2_dashboardでl_elbow_flex_motorのDropsを確認したが0だった.
一時的な解決
upstartのrobotを立ち上げ直したが改善しなかった.(因果関係は定かでないが)左腕を少し動かしてサーボを何度か入れてみたがサーボが上手くはいらずガクガクしている状態だった.しばらく放置して確認作業をしている間にサーボが復活していた.
復活した後のrx_errorのcsvを見ると,l_elbow_flex_motorのRX Error Port 1の値は固定値で上がっていない,やはりこれが今回の原因なのか?
今後の対策
l_elbow_flex_motor
周りのケーブル交換等が必要?