search

robotology / icub-tech-support

Virtual repository that provides support requests for individual robots
GNU General Public License v2.0
20 stars 2 forks source link

Robot not starting after mounting back the forearms #1513

Closed HosameldinMohamed closed 1 year ago

HosameldinMohamed commented 1 year ago

Device name 🤖

iCubGenova09

Request/Failure description

The forearms were removed in #1504 And some ethernet connectors were modified to bypass the forearms in #1506

We mounted back the forearms and reconnected the power and the ethernet but the robot didn't start..

The error is a timeout error for the forearm boards 10.0.1.27, 10.0.1.28, 10.0.1.29, and 10.0.1.24, 10.0.1.25, 10.0.1.26.

Detailed context

Example logs of a good board:

[DEBUG] eth::parser::print(boardData) for BOARD right_arm-eb4-j2_3
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES:
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/IpAddress =  10.0.1.4
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/IpPort =  12345
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/Type =  ems4
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/maxSizeRXpacket =  768
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/maxSizeROP =  384
[DEBUG] ETH_BOARD/ETH_BOARD_SETTINGS:
[DEBUG] ETH_BOARD/ETH_BOARD_SETTINGS/Name =  right_arm-eb4-j2_3
[DEBUG] ETH_BOARD/ETH_BOARD_SETTINGS/RUNNINGMODE/(period, maxTimeOfRXactivity, maxTimeOfDOactivity, maxTimeOfTXactivity, TXrateOfRegularROPs) =  1000 400 300 300 5
[DEBUG] ETH_BOARD/ETH_BOARD_ACTIONS/MONITOR_ITS_PRESENCE
[DEBUG] ETH_BOARD/ETH_BOARD_ACTIONS/MONITOR_ITS_PRESENCE/(enabled, timeout, periodOfMissingReport) =  true 0.02 60
[DEBUG] eth::EthMonitorPresence::config(): monitoring of presence is ON for BOARD 10.0.1.4 (right_arm-eb4-j2_3) with timeout = 0.02 sec and period of missing report = 60 sec
[DEBUG] TheEthManager::requestResource2(): has just succesfully created a new EthResource for board of type ems4 with IP =  10.0.1.4
[WARNING] embObjMC BOARD  right_arm-eb4-j2_3 Missing KALMAN_FILTER group. Kalman Filter will be disabled by default.
[WARNING] viscousPos parameter not found for board right_arm-eb4-j2_3 in bottle TRQ_PID_DEFAULT (controlLaw torque) (outputType pwm) (fbkControlUnits metric_units) (outputControlUnits dutycycle_percent) (kp -2.5 3) (kd 0 0) (ki -3.5 4) (maxOutput 25 25) (maxInt 1.56000000000000005329 1.56000000000000005329) (ko 0 0) (stictionUp 0 0) (stictionDown 0 0) (kff 1 1) (kbemf -0.00300000000000000006245 0.00229999999999999996114) (filterType 0 0) (ktau -0.900000000000000022204 0.699999999999999955591)
[WARNING] viscousNeg parameter not found for board right_arm-eb4-j2_3 in bottle TRQ_PID_DEFAULT (controlLaw torque) (outputType pwm) (fbkControlUnits metric_units) (outputControlUnits dutycycle_percent) (kp -2.5 3) (kd 0 0) (ki -3.5 4) (maxOutput 25 25) (maxInt 1.56000000000000005329 1.56000000000000005329) (ko 0 0) (stictionUp 0 0) (stictionDown 0 0) (kff 1 1) (kbemf -0.00300000000000000006245 0.00229999999999999996114) (filterType 0 0) (ktau -0.900000000000000022204 0.699999999999999955591)
[WARNING] coulombPos parameter not found for board right_arm-eb4-j2_3 in bottle TRQ_PID_DEFAULT (controlLaw torque) (outputType pwm) (fbkControlUnits metric_units) (outputControlUnits dutycycle_percent) (kp -2.5 3) (kd 0 0) (ki -3.5 4) (maxOutput 25 25) (maxInt 1.56000000000000005329 1.56000000000000005329) (ko 0 0) (stictionUp 0 0) (stictionDown 0 0) (kff 1 1) (kbemf -0.00300000000000000006245 0.00229999999999999996114) (filterType 0 0) (ktau -0.900000000000000022204 0.699999999999999955591)
[WARNING] coulombNeg parameter not found for board right_arm-eb4-j2_3 in bottle TRQ_PID_DEFAULT (controlLaw torque) (outputType pwm) (fbkControlUnits metric_units) (outputControlUnits dutycycle_percent) (kp -2.5 3) (kd 0 0) (ki -3.5 4) (maxOutput 25 25) (maxInt 1.56000000000000005329 1.56000000000000005329) (ko 0 0) (stictionUp 0 0) (stictionDown 0 0) (kff 1 1) (kbemf -0.00300000000000000006245 0.00229999999999999996114) (filterType 0 0) (ktau -0.900000000000000022204 0.699999999999999955591)
[INFO]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 0, time 1948s 852m 528u: (code 0x00000034, par16 0x0001 par64 0x0000000000000014) -> SYS: the board has detected a restart of the remote transceiver because the rx ropframe sequence number is 1. In par64 there is the expected number. + .
[DEBUG] EthResource::verifyBoardPresence() found BOARD right_arm-eb4-j2_3 with IP 10.0.1.4 after 0.00301504 seconds
[DEBUG] EthResource::verifyBoardTransceiver() has validated the transceiver of BOARD right_arm-eb4-j2_3 with IP 10.0.1.4
[DEBUG] EthResource::cleanBoardBehaviour() has cleaned the application in BOARD right_arm-eb4-j2_3 with IP 10.0.1.4 : config mode + cleared all its regulars
[DEBUG] EthResource::setTimingOfRunningCycle() for BOARD right_arm-eb4-j2_3 with IP 10.0.1.4 has succesfully set: cycletime = 1000 usec, RX DO TX = ( 400 300 300 ) usec and TX rate = 5  every cycle
[INFO] EthResource::askBoardVersion() found BOARD right_arm-eb4-j2_3 @ IP 10.0.1.4 of type ems4 with FW = ver 3.61 built on 2023 Feb 21 14:30
[DEBUG]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 0, time 1948s 894m 382u: (code 0x05000015, par16 0x2000 par64 0x0000000000ff0000) -> CFG: EOtheEncoderReader can be correctly configured. p16&0xf000: number of joint; primary encs: failure mask in p16&0x000f and errorcodes in p64&0x0000ffff; secondary encs: failure mask in p16&0x00f0 and errorcodes in p64&0xffff0000 + .
[INFO]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3) @ 1948s 894m 579u: CAN discovery has started for 2 eobrd_foc boards on (can1map, can2map) = (0x0018, 0x0000) with target can protocol ver 1.6 and application ver 3.3.3.
[WARNING]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3) @ 1948s 895m 158u: CAN discovery has detected a eobrd_foc board in CAN1 addr 3 with can protocol ver 1.6 and application ver 3.3.12 Search time was 0 ms
[WARNING]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3) @ 1948s 895m 313u: CAN discovery has detected a eobrd_foc board in CAN1 addr 4 with can protocol ver 1.6 and application ver 3.3.12 Search time was 0 ms
[INFO]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3) @ 1948s 895m 475u: CAN discovery is OK for 2 eobrd_foc boards with target can protocol ver 1.6 and application ver 3.3.3. Search time was 0 ms
[DEBUG]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 0, time 1948s 895m 855u: (code 0x0500000d, par16 0x0000 par64 0x0000000000000000) -> CFG: EOtheMotionController can correctly configure 2foc-based motion. more info will follow + .
[DEBUG] embObjMotionControl:serviceVerifyActivate OK!
[DEBUG]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 0, time 1948s 940m 196u: (code 0x04000001, par16 0x000f par64 0x0000010000000040) -> DEBUG: tag01 + CURRENT PID
[DEBUG]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 0, time 1948s 940m 310u: (code 0x04000001, par16 0x000f par64 0x0000018000000200) -> DEBUG: tag01 + VELOCITY PID
[DEBUG]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 1, time 1948s 952m 395u: (code 0x04000001, par16 0x000f par64 0x0000010000000040) -> DEBUG: tag01 + CURRENT PID
[DEBUG]  from BOARD 10.0.1.4 (right_arm-eb4-j2_3), src LOCAL, adr 1, time 1948s 952m 510u: (code 0x04000001, par16 0x000f par64 0x0000018000000200) -> DEBUG: tag01 + VELOCITY PID

Example logs of the faulty boards:

[DEBUG] eth::parser::print(boardData) for BOARD right_arm-eb27-j4_7
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES:
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/IpAddress =  10.0.1.27
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/IpPort =  12345
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/Type =  mc4plus
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/maxSizeRXpacket =  768
[DEBUG] ETH_BOARD/ETH_BOARD_PROPERTIES/maxSizeROP =  384
[DEBUG] ETH_BOARD/ETH_BOARD_SETTINGS:
[DEBUG] ETH_BOARD/ETH_BOARD_SETTINGS/Name =  right_arm-eb27-j4_7
[DEBUG] ETH_BOARD/ETH_BOARD_SETTINGS/RUNNINGMODE/(period, maxTimeOfRXactivity, maxTimeOfDOactivity, maxTimeOfTXactivity, TXrateOfRegularROPs) =  1000 400 300 300 5
[DEBUG] ETH_BOARD/ETH_BOARD_ACTIONS/MONITOR_ITS_PRESENCE
[DEBUG] ETH_BOARD/ETH_BOARD_ACTIONS/MONITOR_ITS_PRESENCE/(enabled, timeout, periodOfMissingReport) =  true 0.02 60
[DEBUG] eth::EthMonitorPresence::config(): monitoring of presence is ON for BOARD 10.0.1.27 (right_arm-eb27-j4_7) with timeout = 0.02 sec and period of missing report = 60 sec
[DEBUG] TheEthManager::requestResource2(): has just succesfully created a new EthResource for board of type mc4plus with IP =  10.0.1.27
[WARNING] embObjMC BOARD  right_arm-eb27-j4_7 Missing OTHER_CONTROL_PARAMETERS.DeadZone parameter. I'll use default value. (see documentation for more datails)
[WARNING] embObjMC BOARD  right_arm-eb27-j4_7 Missing KALMAN_FILTER group. Kalman Filter will be disabled by default.
[INFO]  from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src LOCAL, adr 0, time 1949s 613m 982u: (code 0x00000034, par16 0x0001 par64 0x0000000000000008) -> SYS: the board has detected a restart of the remote transceiver because the rx ropframe sequence number is 1. In par64 there is the expected number. + .
[DEBUG] EthResource::verifyBoardPresence() found BOARD right_arm-eb27-j4_7 with IP 10.0.1.27 after 0.00310826 seconds
[DEBUG] EthResource::verifyBoardTransceiver() has validated the transceiver of BOARD right_arm-eb27-j4_7 with IP 10.0.1.27
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD right_arm-eb27-j4_7 IP 10.0.1.27 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.27 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD right_arm-eb27-j4_7 IP 10.0.1.27 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.27 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD right_arm-eb27-j4_7 IP 10.0.1.27 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.27 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] EthResource::serviceCommand() failed an acked activation request to BOARD right_arm-eb27-j4_7 with IP 10.0.1.27 after 3 attempts each with waiting timeout of 0.5 seconds
[ERROR] EthResource::cleanBoardBehaviour() cannot stop services for BOARD right_arm-eb27-j4_7 with IP 10.0.1.27 : cannot proceed any further
[ERROR] EthResource::verifyEPprotocol() cannot verify BOARD right_arm-eb27-j4_7 with IP 10.0.1.27 : cannot proceed any further
[ERROR] embObjMotionControl: failed verifyEPprotocol. Cannot continue!
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD right_arm-eb27-j4_7 IP 10.0.1.27 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.27 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD right_arm-eb27-j4_7 IP 10.0.1.27 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.27 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD right_arm-eb27-j4_7 IP 10.0.1.27 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.27 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] EthResource::serviceCommand() failed an acked activation request to BOARD right_arm-eb27-j4_7 with IP 10.0.1.27 after 3 attempts each with waiting timeout of 0.5 seconds
[ERROR] |yarp.dev.PolyDriver|right_arm-eb27-j4_7-mc| Driver <embObjMotionControl> was found but could not open

Additional context

No response

How does it affect you?

It blocks tests of colleagues who need the full robot functioning.

The robot can start with iRonCub config files (bypassing forearms).

sgiraz commented 1 year ago

After a FW update of all the boards,the robots started correctly!

HosameldinMohamed commented 1 year ago

Thanks @sgiraz for saving the day!

Maybe the error messages can be updated to include the firmware mismatch? (not sure which device was that)..

sgiraz commented 1 year ago

Hi @HosameldinMohamed, in icub-main there are multiple checks on possible FW versions mismatch (search for strings line "need a FW update"). Actually, I don't remember the whole log of the yarprobotinterface, but I don't remember seeing such strings even if they should be there.

Anyway, we are already aware that diagnostics need to be improved in general.