`Base Fan Off` error after stopping servo

[furushchev]

After cycling servo off -> on using e-stop, the servo is sometimes not turned on. Looking at rqt_pr2_dashboard, it looks the error comes from Base Fan Off Error of Power Board. (I see once each with PR1012 and PR1040) However, it sounds that the base fan was working (I can hear the sound of the fan). I tried to 1. cycling servo off -> on using e-stop again, 2. rebooting C1/C2 but still no luck. After cold rebooting with the red toggle switch, the error is no more displayed and all servos work. (the base fan also works without an error). As far as I see the detailed diagnostic message for /Power System/Power board 0 section, it looks that the system could not get any information from the power board and in the case, the Base Fan Off message is displayed. (I mean the base fan was actually working)

Full Name: /Power System/Power board 0
Component: Power board 0
Hardware ID: 680500700
Level: ERROR
Message: Base Fan Off

Serial Number: 0
Input Current: 0
Input Voltage: 0
DCDC 12V aux: 0
DCDC 12V c1: 0
Breaker 0 Voltage: 0
Breaker 1 Voltage: 0
Breaker 2 Voltage: 0
Board Temperature: 0
Base Fan Speed: 0
Expansion Fan 1 Speed: 0
Expansion Fan 2 Speed: 0
Expansion Fan 3 Speed: 0
Breaker 0 (Left Arm) State: no-power
Breaker 1 (Base/Body) State: no-power
Breaker 2 (Right Arm) State: no-power
DCDC state: no-power
RunStop Button Status: False
RunStop Wireless Status: False
Circuit assembly revision: 
Circuit board revision: 
Major Revision: 
Minor Revision: 
Min Voltage: 0
Max Current: 0
DCDC 12V c2: 0
DCDC 12V user: 0
Battery 0 voltage=: 0
Battery 1 voltage=: 0
Battery 2 voltage=: 0
Battery 3 voltage=: 0
CB0 Stop Count: 0
CB0 E-Stop Count: 0
CB0 Trip Count: 0
CB0 18V Fail Count: 0
CB0 Disable Count: 0
CB0 Start Count: 0
CB0 Pump Fail Count: 0
CB0 Reset Count: 0
CB1 Stop Count: 0
CB1 E-Stop Count: 0
CB1 Trip Count: 0
CB1 18V Fail Count: 0
CB1 Disable Count: 0
CB1 Start Count: 0
CB1 Pump Fail Count: 0
CB1 Reset Count: 0
CB2 Stop Count: 0
CB2 E-Stop Count: 0
CB2 Trip Count: 0
CB2 18V Fail Count: 0
CB2 Disable Count: 0
CB2 Start Count: 0
CB2 Pump Fail Count: 0
CB2 Reset Count: 0

[furushchev]

GA Tech guys also reported the similar symptom on their wiki: http://hrl-wiki.gatech.edu/hrl-wiki/index.php?title=PR2_Problems

[furushchev]

rostopic echo /power_board/state also does not publish anything.

(It should publish periodic messages like below)

header: 
  seq: 612283
  stamp: 
    secs: 1533734236
    nsecs: 232751122
  frame_id: ''
name: Power board 1085
serial_num: 1085
input_voltage: 71.1439056396
master_state: 2
circuit_state: [1, 1, 1]
circuit_voltage: [17.441322326660156, 17.390731811523438, 17.302196502685547]
run_stop: False
wireless_stop: False
---

[furushchev]

Sniffing packets from the power board, periodic data is captured on PR1040 (it works without error)

pr2admin@pr1040:~$ sudo ngrep src 10.68.0.50 -W byline
interface: lan0 (10.68.0.0/255.255.255.0)
filter: (ip or ip6) and ( src 10.68.0.50 )
#
U 10.68.0.50:6802 -> 10.68.0.1:6801
....=...PR2 Power status message............].......4C.B..}@".?A..HA...A8 .A.j.A...B.....................1D48&=.B...@.WHA....)3.A*..A7..A89.A
#
U 10.68.0.50:6802 -> 10.68.0.1:6801
....=...PR2 Power status message............]........j.B1wt@..?A..HA.:.A8 .A.j.A...B.....................1D48&=.B...@..HA.....D.A...AKJ.A.d.A
#
U 10.68.0.50:6802 -> 10.68.0.1:6801
....=...PR2 Power status message............]........V.B..u@".?A..HA.:.Ai..A.j.A...B.....................1D48&=.B...@..HA....),.A...A...A...A
#
U 10.68.0.50:6802 -> 10.68.0.1:6801
....=...PR2 Power status message............].......@6.B.\l@,.?A..HA.:.Ai..A.j.A...B.....................1D48&=.B...@..HA=..;x..A.?.A...A^..A
#
U 10.68.0.50:6802 -> 10.68.0.1:6801
....=...PR2 Power status message............]..........B..m@..?A..HA.T.AQ..A.j.A...B.....................1D48&=.B...@..HA.....$.A|..A...A_u.A
#

But there is no data received on PR1012:

pr2admin@pr1012:~$ sudo ngrep src 10.68.0.50 -W byline
[sudo] password for pr2admin: 
interface: lan0 (10.68.0.0/255.255.255.0)
filter: (ip or ip6) and ( src 10.68.0.50 )
^Cexit
11 received, 0 dropped

The possible causes will be:

• The power board transited into an unexpected state (on network)
• Failure on initialization
• Network problem (between the power board and the computer)