r_shoulder_yaw goes in hardware fault during yoga++

[traversaro]

Description of the failure

While running the YOGAPP, the joint 2 of the right_arm part (r_shoulder_yaw) went in hardware fault.

Details

40505       74136,811236        ERROR        from BOARD 10.0.1.6 (EB6), src CAN2, adr 13, time 510s 822m 91u: (code 0x01000002, par16 0x0003 par64 0x0000000d00000000) -> HW: strain values saturation. In p16 there is the channel involved, in p64 there are the saturation counts for the last sec of both types (lower: LS32bits, upper MS32bits + .
40508       74136,811390        ERROR        from BOARD 10.0.1.6 (EB6), src CAN2, adr 13, time 510s 822m 104u: (code 0x01000002, par16 0x0004 par64 0x0000000000000021) -> HW: strain values saturation. In p16 there is the channel involved, in p64 there are the saturation counts for the last sec of both types (lower: LS32bits, upper MS32bits + .
40540       74150,270022        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 524s 294m 408u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
40553       74155,278112        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 529s 294m 456u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
40568       74160,273503        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 534s 295m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
40580       74165,270521        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 539s 296m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
40586       74170,276884        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 544s 297m 411u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
40604       74175,276448        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 549s 298m 411u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
40619       74176,924115        ERROR        from BOARD 10.0.1.6 (EB6), src CAN2, adr 13, time 550s 932m 158u: (code 0x01000002, par16 0x0003 par64 0x000001fa00000000) -> HW: strain values saturation. In p16 there is the channel involved, in p64 there are the saturation counts for the last sec of both types (lower: LS32bits, upper MS32bits + .

[francesco-romano]

How many times it happened? Do you also have warnings about spikes in the encoder just before the fault?

I had the same issue, but because it happened during a tendon breakage, I though it was related to that.

[traversaro]

42210       74654,471948        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1028s 490m 407u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42229       74657,147241        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1031s 167m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42235       74662,145164        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1036s 168m 449u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42248       74667,152651        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1041s 169m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42250       74668,283576        ERROR        from BOARD 10.0.1.6 (EB6), src CAN2, adr 13, time 1042s 281m 83u: (code 0x01000002, par16 0x0003 par64 0x0000000d00000000) -> HW: strain values saturation. In p16 there is the channel involved, in p64 there are the saturation counts for the last sec of both types (lower: LS32bits, upper MS32bits + .
42267       74672,148777        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1046s 170m 414u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42318       74675,583524        ERROR       A 0.250000 sec timeout occured in embObjMotionControl::checkRemoteControlModeStatus(), BOARD EB3 IP 10.0.1.3, joint 0, current mode: idl, requested: pos
42320       74675,583564        ERROR       failure of embObjMotionControl::checkRemoteControlModeStatus(j=0, targetmode=pos) for BOARD EB3 IP 10.0.1.3 after 17 attempts and 0.264333 seconds
42323       74675,593617        ERROR       In embObjMotionControl::setControlModeRaw(j=0, mode=pos) for BOARD EB3 IP 10.0.1.3 has failed checkRemoteControlModeStatus()
42355       74675,845737        ERROR       A 0.250000 sec timeout occured in embObjMotionControl::checkRemoteControlModeStatus(), BOARD EB3 IP 10.0.1.3, joint 1, current mode: idl, requested: pos
42357       74675,845752        ERROR       failure of embObjMotionControl::checkRemoteControlModeStatus(j=1, targetmode=pos) for BOARD EB3 IP 10.0.1.3 after 16 attempts and 0.252570 seconds
42359       74675,845764        ERROR       In embObjMotionControl::setControlModeRaw(j=1, mode=pos) for BOARD EB3 IP 10.0.1.3 has failed checkRemoteControlModeStatus()
42361       74675,845777        ERROR       embObjMotionControl::checkRemoteControlModeStatus(2, 7565168) is unable to check the control mode of BOARD EB3 IP 10.0.1.3 because it is now in HW_FAULT
42378       74677,147089        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1051s 171m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42396       74678,307017        ERROR        from BOARD 10.0.1.6 (EB6), src CAN2, adr 13, time 1052s 303m 93u: (code 0x01000002, par16 0x0003 par64 0x000001a700000000) -> HW: strain values saturation. In p16 there is the channel involved, in p64 there are the saturation counts for the last sec of both types (lower: LS32bits, upper MS32bits + .
42398       74678,307035        ERROR        from BOARD 10.0.1.6 (EB6), src CAN2, adr 13, time 1052s 303m 104u: (code 0x01000002, par16 0x0004 par64 0x0000000000000121) -> HW: strain values saturation. In p16 there is the channel involved, in p64 there are the saturation counts for the last sec of both types (lower: LS32bits, upper MS32bits + .
42426       74682,149535        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1056s 172m 411u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42443       74687,153482        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1061s 173m 450u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42455       74692,153680        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1066s 174m 419u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42468       74697,156342        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1071s 175m 450u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42486       74702,159995        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1076s 176m 449u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42506       74707,154209        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1081s 177m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42524       74712,155936        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1086s 178m 411u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42542       74717,160729        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1091s 179m 411u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42548       74722,164336        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1096s 180m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42565       74727,159399        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1101s 181m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42581       74732,161979        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1106s 182m 450u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42601       74737,166308        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1111s 183m 411u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42613       74742,156941        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1116s 184m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42631       74747,158195        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1121s 185m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42640       74752,160909        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1126s 186m 413u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42654       74757,164795        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1131s 187m 414u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
42669       74762,166774        ERROR        from BOARD 10.0.1.3 (EB3), src LOCAL, adr 0, time 1136s 188m 449u: (code 0x02000008, par16 0x0002 par64 0x0000000000000009) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .

[traversaro]

It happened again. The interesting thing is that the AEA encoder invalid data error is reported exactly every 5 seconds when the joint is in hardware fault, but when we send a FORCE_IDLE message and the joint goes back in idle mode and then in position, the error stops.

[DanielePucci]

It seems that we have a problem with the encoder of the right shoulder, joint 2 i.e.

[DanielePucci]

@julijenv

[DanielePucci]

Suspicious values of the AEA JTN 2 right_shoulder occurred again. @spada1978 and Gandini are replacing it.

[spada1978]

replaced AEA reached n.2