ergoCub 1.1 S/N:001 – The robot does not re-start after a stop w/ CTRL+C

[xEnVrE]

Robot Name 🤖

ergoCub 1.1 S/N:001

Request/Failure description

We were testing things on the robot and we had to restart it at some point to run using a different configuration file (that was in addition adding support for publishing joints on ROS 2).

At some point the robot stopped starting for reasons that are not really very clear. See the "Detailed context" for more information.

Detailed context

The robot was started from the ergocug-torso machine using yarprobotinterface --config ergocub_all.xml, where the configuration file is the one inside the Antonio/devel branch - as this is the one that I found.

I run from the CLI directly and this is the dumped log from the yarprobotinterface: log.txt

I checked it already together with @AntonioConsilvio and the guess is that this line

embObjMotionControl::open() fails to start mc service for BOARD left_arm-eb31-j4_6 (IP 10.0.1.31)  : cannot continue

is probably indicating a possible issue.

Additional context

No response

How does it affect you?

No response

[sgiraz]

Hi @xEnVrE,

Could you post here the result of FirmwareUpdater after the discovery of ETH board, pls? I suppose that the board eb31 does not start in eApplication, but it is stuck in eUpdater. In that case, it should be enough click on Set Default Boot Application button

cc @AntonioConsilvio

[Nicogene]

cc @PasMarra

[SimoneMic]

Hi @sgiraz, Nicola is not working in IIT anymore, so when we will use the robot we will look at the FW updater if the issue persists.

[AntonioConsilvio]

Hi @SimoneMic, I removed the forearm cover and I found a damaged CAN cable (31N1) that connect AMCBLDC (31B1) to AMC (EB31).

The damaged cable caused the inability to find the AMCBLD board and therefore the closure of the yarprobotinterface.

I fixed It and the robot is working properly now! However, feel free to reopen the issue if the problem happens again.

cc @sgiraz

[SimoneMic]

Hello,

Yesterday we used the robot and during a restart of the yarprobotinterface we encountered again the mentioned issue. After restarting the motors the problem disappeared and wasn't re-encountered.

Here's the log, the CAN board is still the same BOARD 10.0.1.31 - left_arm-eb31-j4_6

[DEBUG] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8723s 768m 216u : CFG: EOtheEncoderReader can be correctly configured
[INFO] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8723s 768m 294u :  CFG: CANdiscovery started the search for 3 eobrd_amcbldc boards on (can1map, can2map) = (0x000e, 0x0000) with target can protocol ver 2.0 and application ver 2.0.4.
[ERROR] theNVmanager::Impl::wait() had a timeout for BOARD left_arm-eb31-j4_6 IP 10.0.1.31 and nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[ERROR] theNVmanager::Impl::command() fails a wait() from IP 10.0.1.31 for nv ID32 = 0x00030001 -> IND = 0, TAG = eoprot_tag_mn_service_status_commandresult
[DEBUG] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 114u : CFG: EOtheEncoderReader can be correctly configured
[DEBUG] embObjMotionControl:serviceVerifyActivate OK!
[INFO] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 199u : CFG: CANdiscovery has detected  eobrd_amcbldc board in CAN1 addr 1 with can protocol ver 2.0 and application ver 2.0.4 Search time was 0 ms
[INFO] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 270u : CFG: CANdiscovery has detected  eobrd_amcbldc board in CAN1 addr 2 with can protocol ver 2.0 and application ver 2.0.4 Search time was 0 ms
[ERROR] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 345u : CFG: CANdiscovery cannot find  1 missing eobrd_amcbldc boards for 500 ms in CAN1:1 of 1: missing eobrd_amcbldc BOARD 10.0.1.31:CAN1:3
[ERROR] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 433u : CFG: EOtheMotionController cannot be configured. CANdiscovery of 2foc boards fails. see CANdiscovery messages for more details
[INFO] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 581u :  CFG: CANdiscovery started the search for 3 eobrd_amcbldc boards on (can1map, can2map) = (0x000e, 0x0000) with target can protocol ver 2.0 and application ver 2.0.4.`

Unfortunately I cannot re-open the issue

cc @PasMarra cc @AntonioConsilvio

[sgiraz]

Ok guys, this is the most important information:

[ERROR] from BOARD 10.0.1.31 (left_arm-eb31-j4_6) time=8724s 269m 345u : CFG: CANdiscovery cannot find 1 missing eobrd_amcbldc boards for 500 ms in CAN1:1 of 1: missing eobrd_amcbldc BOARD 10.0.1.31:CAN1:3

Actually, on this robot, the eb31 CAN1:3 is dedicated to the amc2c (2nd core of the amc board). I would double-check the CAN wiring. If the wiring is ok, we'll have to investigate the problem internally.

[!Note] Consider that the amc2c has been released recently and it is frequently under development. Hence I suggest you update the FW of both amc2c and amcbldc boards to the latest versions (https://github.com/robotology/icub-firmware-build/tree/devel)

cc @AntonioConsilvio

[sgiraz]

Hi guys, is the problem still occurring?

I know that the FW of the boards has been updated. Maybe the SW as well...

cc @AntonioConsilvio @SimoneMic

[SimoneMic]

Hello @sgiraz

This issue is still occurring consistently, once we stop the yarprobotinterface and restart it afterwards, the error reappears (without turning off the motors).

[sgiraz]

Hi @SimoneMic,

Since the porta a porta demo and after https://github.com/robotology/icub-firmware/pull/458 and https://github.com/robotology/icub-firmware/pull/459, the problem didn't happen anymore. Moreover many tests have been conducted during the last days e none experienced this problem anymore.

If you are agree, I would proceed to close this issue.

cc @AntonioConsilvio @PasMarra

[SimoneMic]

Yes we can close this

[sgiraz]

Ok, thanks for your feeedback!

Closing!