Servo sudden turns off

[furushchev]

Maybe back left base motors failed?

[takayuki5168]

Motors of PR1040 died frequently.

cc @furushchev

[furushchev]

@takayuki5168 Thx for report! I further investigated a bit more. Comparing the suspective motor and normal one:

# fl_caster_rotation_motor
Full Name: /Motors/EtherCAT Device (fl_caster_rotation_motor)
Component: EtherCAT Device (fl_caster_rotation_motor)
Hardware ID: 68-05005-02733
Level: OK
Message: OK

Configuration: good
Name: fl_caster_rotation_motor
Position: 07
Product code: WG05 (6805005) Firmware Revision 1.22, PCB Revision F.03
Robot: PR2
Motor: Maxon 236672
Serial Number: 68-05005-02733
Nominal Current Scale: 0.000500
Nominal Voltage Scale: 0.006250
HW Max Current: 3.250000
SW Max Current: 0.655000
Speed Constant: 136.000000
Resistance: 16.700000
Motor Torque Constant: 0.070400
Pulses Per Revolution: 1200
Encoder Reduction: -1.000000
Status Checksum Error Count: 0
Safety Disable Status: ENABLED (00)
Safety Disable Status Hold: ENABLED (00)
Safety Disable Count: 48
Undervoltage Count: 25
Over Current Count: 0
Board Over Temp Count: 0
Bridge Over Temp Count: 0
Operate Disable Count: 1
Watchdog Disable Count: 24
MBX Command IRQ Count: 173
PDI Timeout Error Count: 0
PDI Checksum Error Count: 0
Supply Current: 0.044651
Configured Offset A: 0.011000
Configured Offset B: 0.001500
Mailbox Write Errors: 0
Mailbox Read Errors: 0
Mailbox Retries: 0
Mailbox Retry Errors: 0
Calibration Offset: 61.192989
Calibration Status: Using saved calibration
Watchdog Limit: 100ms
Mode: ENABLE, CURRENT
Digital out: 254
Programmed pwm value: 128
Programmed current: 0.029000
Measured current: 0.029000
Timestamp: 2326156900
Encoder count: 332626
Encoder index pos: 332626
Num encoder_errors: 0
Encoder status: 13
Calibration reading: 14
Last calibration rising edge: 344442
Last calibration falling edge: 344248
Board temperature: 51.312500
Max board temperature: 61.500000
Bridge temperature: 55.000000
Max bridge temperature: 65.437500
Supply voltage: 70.887505
Motor voltage: 0.343750
Current Loop Kp: 16
Current Loop Ki: 4
Motor Voltage Error %: -0.758639
Max Motor Voltage Error %: 26.233989
Abs Filtered Voltage Error %: 0.823610
Max Abs Filtered Voltage Error %: 15.789053
Current Error: -0.000036
Max Current Error: 0.255035
Abs Filtered Current Error: 0.000045
Max Abs Filtered Current Error: 0.078921
Motor Resistance Estimate: 17.766763
# Published traces: 3
Packet count: 43269
Drops: 16542
Consecutive Drops: 0
Max Consecutive Drops: 1873
Reset detected: No
Valid: Yes
EPU Errors: 89
PDI Errors: 0
Status Port 0: Has Link, Open, Has Comm
RX Error Port 0: 0
Forwarded RX Error Port 0: 87
Invalid Frame Port 0: 2
Lost Link Port 0: 0
Status Port 1: No Link, Closed, No Comm
RX Error Port 1: 0
Forwarded RX Error Port 1: 0
Invalid Frame Port 1: 0
Lost Link Port 1: 0

# fr_caster_rotation_motor
Full Name: /Motors/EtherCAT Device (fr_caster_rotation_motor)
Component: EtherCAT Device (fr_caster_rotation_motor)
Hardware ID: 68-05005-02765
Level: OK
Message: OK

Configuration: good
Name: fr_caster_rotation_motor
Position: 13
Product code: WG05 (6805005) Firmware Revision 1.22, PCB Revision F.03
Robot: PR2
Motor: Maxon 236672
Serial Number: 68-05005-02765
Nominal Current Scale: 0.000500
Nominal Voltage Scale: 0.006250
HW Max Current: 3.250000
SW Max Current: 0.655000
Speed Constant: 136.000000
Resistance: 16.700000
Motor Torque Constant: 0.070400
Pulses Per Revolution: 1200
Encoder Reduction: -1.000000
Status Checksum Error Count: 0
Safety Disable Status: ENABLED (00)
Safety Disable Status Hold: ENABLED (00)
Safety Disable Count: 32
Undervoltage Count: 29
Over Current Count: 0
Board Over Temp Count: 0
Bridge Over Temp Count: 0
Operate Disable Count: 1
Watchdog Disable Count: 4
MBX Command IRQ Count: 249
PDI Timeout Error Count: 0
PDI Checksum Error Count: 0
Supply Current: 0.043302
Configured Offset A: -0.000500
Configured Offset B: 0.009500
Mailbox Write Errors: 0
Mailbox Read Errors: 0
Mailbox Retries: 0
Mailbox Retry Errors: 0
Calibration Offset: 60.559434
Calibration Status: Using saved calibration
Watchdog Limit: 100ms
Mode: ENABLE, CURRENT
Digital out: 254
Programmed pwm value: -63
Programmed current: -0.017000
Measured current: -0.017000
Timestamp: 2347601700
Encoder count: 237380
Encoder index pos: 237380
Num encoder_errors: 0
Encoder status: 15
Calibration reading: 14
Last calibration rising edge: 201496
Last calibration falling edge: 201491
Board temperature: 49.625000
Max board temperature: 56.500000
Bridge temperature: 53.562500
Max bridge temperature: 60.687500
Supply voltage: 70.800003
Motor voltage: -0.368750
Current Loop Kp: 16
Current Loop Ki: 4
Motor Voltage Error %: -0.431028
Max Motor Voltage Error %: 30.459609
Abs Filtered Voltage Error %: 0.490277
Max Abs Filtered Voltage Error %: 19.864354
Current Error: 0.000001
Max Current Error: 0.264661
Abs Filtered Current Error: 0.000028
Max Abs Filtered Current Error: 0.082818
Motor Resistance Estimate: 17.853722
# Published traces: 7
Packet count: 19780
Drops: 0
Consecutive Drops: 0
Max Consecutive Drops: 0
Reset detected: No
Valid: Yes
EPU Errors: 150
PDI Errors: 0
Status Port 0: Has Link, Open, Has Comm
RX Error Port 0: 0
Forwarded RX Error Port 0: 148
Invalid Frame Port 0: 2
Lost Link Port 0: 0
Status Port 1: No Link, Closed, No Comm
RX Error Port 1: 0
Forwarded RX Error Port 1: 0
Invalid Frame Port 1: 0
Lost Link Port 1: 0

Drops counts much more at front left base module than at front right. I suspect the communication cable or port between ECAT master and front left motor is going to be dying.

[furushchev]

Since 3 of 3 motors of front left casters fails at the same time, I suspect the cable is dead. I replaced the cable with new one but still no luck.

[furushchev]

Maybe the port is dead, I'll now try to swap the port that is connected to the front left caster with the other port.

[furushchev]

Swapped the ports connected to ECAT hub, but still the front left caster does not work.

[furushchev]

Possible cause are from the slipring or MCB itself (three MCBs are connected with daisy chaining, so if one MCB fails, it can looks 3 MCBs are dead?). I'll try which is the actual cause by bypassing slipring.

[furushchev]

This issue occured again yesterday, I monitored the motor log and found that the fault occured at the block of fl_caster (front left) as same as the last fault but all three motor boards makes no error in the block (from which I suspect that the fault is from slipring). In a few days, let's replace the slipring and see how it goes. CC: @takayuki5168 @weiqiyang @Affonso-Gui

[furushchev]

We replaced the slip ring and it now works fine. @takayuki5168 @Affonso-Gui Thanks for cooperation!!

Slip ring replacement.pdf