ergoCub 1.0 S/N:000 - The motors LED often blinks red while walking connected to the power supply

S-Dafarra commented 1 year ago

Robot Name 🤖

ergoCub 1.0 S/N:000

Request/Failure description

In the past couple of weeks, it happened a few times that the motor LED suddenly starts blinking red, and the robot collapse.

Detailed context

This behavior seems to happen more often lately, especially when walking. Interestingly, it seems to happen especially when connected via the power cable, while, so far, it never happened while in battery mode.

This behavior seems quite random, and it happened also during demos with visitors.

Additional context

Here a log right before the failure if it can be useful: log_ergocub-torso_yarprobotinterface_3383_demonstration_failure.txt

Since while walking, the robot is position controlled, I wonder whether the problems to the encoders could be the cause (e.g. sudden error on a joint, that then requires a lot of power):

1602
1603

I also wonder if the version of the BAT is correct, or if it is related to the BAT FW. I remember that we already had these issues in the past and that, in the newest revision of the BAT, the shunt resistors were cutting power at much higher current consumption.

cc @DanielePucci, @maggia80, @GiulioRomualdi @mebbaid

How does it affect you?

No response

maggia80 commented 1 year ago

cc: @MSECode

DanielePucci commented 1 year ago

I just add that lately we had demos without the robot protection (crane), so this isssue is a sort of possible "lightening" from the sky that may make demos outdoor fail badly (robot falling on stage, difficult to fix outdoor, etc)

sgiraz commented 1 year ago

Hi guys,

Just FYI, @MSECode is working on the implementation of better diagnostics that should help us to understand the reason for this issue. Actually, it seems the problem occurs when we try to use the robot with current/power limits higher than the standard one, but this is only a conjecture at the moment.

cc @maggia80

GiulioRomualdi commented 1 year ago

On Friday @paolo-viceconte and I tested a new version of the walking-controller and we noticed that the robot was going in overcurrent from time to time both during walking and setting the initial walking configuration with the robot tethered and on the air.

Here are two videos containing the overcurrent while walking

https://github.com/ami-iit/component_bipedal-locomotion/assets/16744101/2305a847-d03b-42d4-9dc3-9f0ebca366d0

https://github.com/ami-iit/component_bipedal-locomotion/assets/16744101/0faa8076-c641-4453-b5f9-4f6582ea599e

For the last overcurrent, we collect also the yarprobotinterface log log_ergocub-torso_yarprobotinterface_27572_overcurrent.txt

Here numerical dataset that may help to better understand the issue overcurrent_datasets.zip

MSECode commented 1 year ago

cc: @MrAndrea

MrAndrea commented 1 year ago

We must check these points:

BAT board installed on the ergoCub should be the iitcode 10050 Rev. D: this revision increases the HSM current limit up to 36A as described in this issue.
The firmware should be the rev. 1.2.0, see this PR.

The firmware revision can be easily verified in the first 5 seconds after robot power up on the display on the back. Instead to check the hardware, we have to remove the cover and see the board.

DanielePucci commented 1 year ago

Thank you all for your concerns @MrAndrea @MSECode.

On September 21st we have an important demo, in front of ministers, at Morego for the IIT birthday https://github.com/ami-iit/lab-events-demos/issues/277

The robot will be without crane, so it would be important for us to have this issue fixed by that day.

Thanks anyways

CC @maggia80

lrapetti commented 1 year ago

Today with @mebbaid we did some experiments testing centroidal MPC walking controller.

TL;DR we observed the issue happening once when connected to battery.

When connected to the power supply we tested the centroidal MPC walking controller multiple times and we observed the following:

the robot was not able to walk (failing after one step)
~50% of the time the robot was switching off with the motor LED blinking red

We then tried a few test connected to the battery:

test 1: test with centroidal MPC the robot was able to walk, no issues.
test 2: test with (modified) centroidal MPC, the robot was not able to walk and we observed again the red blinking LED with the motors switching off.
test 3: test walking controller (not centroidal), we were able to walk around, no issues.

maggia80 commented 1 year ago

@lrapetti could you please log the BMS using the instructions here: https://github.com/robotology/icub-main/tree/devel/src/libraries/icubmod/embObjBattery

S-Dafarra commented 1 year ago

@lrapetti could you please log the BMS using the instructions here: https://github.com/robotology/icub-main/tree/devel/src/libraries/icubmod/embObjBattery

But would this work when the battery LED starts blinking? If I recall correctly, the BMS is connected via CAN to an EMS, but the EMS dies as soon as the triggering event occurs

S-Dafarra commented 1 year ago

Indeed, it should be already enabled: https://github.com/robotology/robots-configuration/commit/2c6dfcc3eaad6398565b066dd7462cd4f25d5b68

lrapetti commented 1 year ago

Indeed, it should be already enabled: robotology/robots-configuration@2c6dfcc

@maggi] I guess then we should only add the port /ergocub/battery/data:o to the logger right, or is there another relevant port to log? I don't know if @GiulioRomualdi can try to log it tomorrow, otherwise, next time we are with the robot we can log it.

maggia80 commented 1 year ago

yes, it's sufficient to add the port to the logger. It's true that when the LED start blinking the boards are switched off and cannot send the last value of the current, but at least we can see the current and the voltage just before the fault. We are implementing a new FW for the BAT to display in the OLED display the fault and the current generating the fault.

maggia80 commented 1 year ago

cc: @MSECode

GiulioRomualdi commented 1 year ago

Hi @MrAndrea and @maggia80 this is the output of the port. At the very end the motors went in overcurrent (we were attached to the battery)

39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 2.90000009536743164062 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.40000152587890625 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 2.90000009536743164062 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 2.90000009536743164062 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 75.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 2.90000009536743164062 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 2.90000009536743164062 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 3.0 76.0 25.0 0
39.5 2.90000009536743164062 76.0 25.0 0
39.09999847412109375 -0.20000000298023223877 76.0 25.0 0
39.09999847412109375 -0.20000000298023223877 76.0 25.0 0
39.09999847412109375 0.0 76.0 25.0 0
39.09999847412109375 0.0 76.0 25.0 0
39.09999847412109375 -0.100000001490116119385 76.0 25.0 0
38.5 -3.79999995231628417969 76.0 25.0 0
38.59999847412109375 -4.09999990463256835938 76.0 25.0 0
38.59999847412109375 -3.90000009536743164062 76.0 25.0 0
38.59999847412109375 -4.19999980926513671875 76.0 25.0 0
38.5 -4.5 76.0 25.0 0
38.59999847412109375 -4.40000009536743164062 76.0 25.0 0
38.59999847412109375 -3.90000009536743164062 76.0 25.0 0
38.40000152587890625 -4.30000019073486328125 76.0 25.0 0
38.59999847412109375 -4.40000009536743164062 76.0 25.0 0
38.59999847412109375 -4.30000019073486328125 76.0 25.0 0
38.59999847412109375 -4.0 76.0 25.0 0
38.299999237060546875 -4.0 76.0 25.0 0
38.299999237060546875 -3.79999995231628417969 76.0 25.0 0
38.40000152587890625 -4.09999990463256835938 76.0 25.0 0
38.59999847412109375 -4.0 76.0 25.0 0
38.40000152587890625 -3.90000009536743164062 76.0 25.0 0
38.59999847412109375 -4.30000019073486328125 76.0 25.0 0
38.5 -4.5 76.0 25.0 0
38.299999237060546875 -3.90000009536743164062 76.0 25.0 0
38.40000152587890625 -3.90000009536743164062 76.0 25.0 0
38.200000762939453125 -4.30000019073486328125 76.0 25.0 0
38.5 -4.19999980926513671875 76.0 25.0 0
38.299999237060546875 -4.5 76.0 25.0 0
38.200000762939453125 -4.09999990463256835938 76.0 25.0 0
38.200000762939453125 -4.40000009536743164062 76.0 25.0 0
38.40000152587890625 -4.19999980926513671875 75.0 25.0 0
38.5 -4.19999980926513671875 75.0 25.0 0
38.200000762939453125 -4.19999980926513671875 75.0 25.0 0
38.5 -3.70000004768371582031 75.0 25.0 0
38.200000762939453125 -4.0 75.0 25.0 0
38.200000762939453125 -3.70000004768371582031 75.0 25.0 0
38.40000152587890625 -4.0 75.0 25.0 0
38.5 -4.0 75.0 25.0 0
38.5 -4.40000009536743164062 75.0 25.0 0
38.09999847412109375 -4.40000009536743164062 75.0 25.0 0
38.200000762939453125 -3.79999995231628417969 75.0 25.0 0
38.200000762939453125 -3.5 75.0 25.0 0
38.200000762939453125 -3.90000009536743164062 75.0 25.0 0
38.40000152587890625 -4.0 75.0 25.0 0
38.09999847412109375 -4.09999990463256835938 75.0 25.0 0
38.40000152587890625 -4.19999980926513671875 75.0 25.0 0
38.40000152587890625 -3.79999995231628417969 75.0 25.0 0
38.40000152587890625 -4.40000009536743164062 75.0 25.0 0
38.40000152587890625 -4.19999980926513671875 75.0 25.0 0
38.299999237060546875 -4.19999980926513671875 75.0 25.0 0
38.09999847412109375 -3.79999995231628417969 75.0 25.0 0
38.200000762939453125 -4.5 75.0 25.0 0
38.200000762939453125 -3.79999995231628417969 75.0 25.0 0
38.5 -3.90000009536743164062 75.0 25.0 0
38.299999237060546875 -4.09999990463256835938 75.0 25.0 0
38.299999237060546875 -4.19999980926513671875 75.0 25.0 0
38.299999237060546875 -4.40000009536743164062 75.0 25.0 0
38.09999847412109375 -4.69999980926513671875 75.0 25.0 0
37.90000152587890625 -5.0 75.0 25.0 0
38.200000762939453125 -4.90000009536743164062 75.0 25.0 0
38.0 -4.59999990463256835938 75.0 25.0 0
38.200000762939453125 -5.40000009536743164062 75.0 25.0 0
38.200000762939453125 -5.09999990463256835938 75.0 25.0 0
38.200000762939453125 -4.90000009536743164062 75.0 25.0 0
38.0 -4.90000009536743164062 75.0 25.0 0
38.0 -5.30000019073486328125 75.0 25.0 0
37.90000152587890625 -5.09999990463256835938 75.0 25.0 0
38.299999237060546875 -5.30000019073486328125 75.0 25.0 0
38.09999847412109375 -5.30000019073486328125 75.0 25.0 0
38.200000762939453125 -5.0 75.0 25.0 0
38.09999847412109375 -5.09999990463256835938 75.0 25.0 0
38.0 -5.30000019073486328125 75.0 25.0 0
37.799999237060546875 -5.0 75.0 25.0 0
38.09999847412109375 -5.0 75.0 25.0 0
38.09999847412109375 -4.69999980926513671875 75.0 25.0 0
38.09999847412109375 -5.30000019073486328125 75.0 25.0 0
38.200000762939453125 -5.19999980926513671875 75.0 25.0 0
37.90000152587890625 -5.09999990463256835938 75.0 25.0 0
38.0 -4.40000009536743164062 75.0 25.0 0
38.200000762939453125 -4.80000019073486328125 75.0 25.0 0
38.0 -5.30000019073486328125 75.0 25.0 0
37.799999237060546875 -4.40000009536743164062 75.0 25.0 0
38.200000762939453125 -4.59999990463256835938 75.0 25.0 0
38.0 -4.69999980926513671875 75.0 25.0 0
38.09999847412109375 -5.09999990463256835938 75.0 25.0 0
37.90000152587890625 -4.59999990463256835938 75.0 25.0 0
37.90000152587890625 -5.09999990463256835938 75.0 25.0 0
37.799999237060546875 -5.0 75.0 25.0 0
37.799999237060546875 -5.0 75.0 25.0 0
38.09999847412109375 -5.5 75.0 25.0 0
38.09999847412109375 -4.90000009536743164062 75.0 25.0 0
38.09999847412109375 -5.40000009536743164062 75.0 25.0 0
37.90000152587890625 -4.80000019073486328125 75.0 25.0 0
37.90000152587890625 -5.40000009536743164062 75.0 25.0 0
38.0 -5.19999980926513671875 75.0 25.0 0
38.09999847412109375 -5.80000019073486328125 75.0 25.0 0
38.09999847412109375 -5.09999990463256835938 75.0 25.0 0
38.09999847412109375 -4.80000019073486328125 75.0 25.0 0
37.799999237060546875 -5.09999990463256835938 75.0 25.0 0
38.09999847412109375 -4.69999980926513671875 75.0 25.0 0
38.09999847412109375 -5.5 74.0 25.0 0
38.0 -4.19999980926513671875 74.0 25.0 0
37.799999237060546875 -4.69999980926513671875 74.0 25.0 0
38.09999847412109375 -4.30000019073486328125 74.0 25.0 0
37.90000152587890625 -4.30000019073486328125 74.0 25.0 0
38.09999847412109375 -4.40000009536743164062 74.0 25.0 0
38.0 -4.80000019073486328125 74.0 25.0 0
38.0 -4.5 74.0 25.0 0
37.90000152587890625 -4.59999990463256835938 74.0 25.0 0
38.0 -4.0 74.0 25.0 0
37.799999237060546875 -4.19999980926513671875 74.0 25.0 0
38.200000762939453125 -4.69999980926513671875 74.0 25.0 0
37.799999237060546875 -4.30000019073486328125 74.0 25.0 0
38.0 -5.09999990463256835938 74.0 25.0 0
38.09999847412109375 -4.69999980926513671875 74.0 25.0 0
38.0 -4.80000019073486328125 74.0 25.0 0
38.09999847412109375 -4.5 74.0 25.0 0
38.09999847412109375 -4.69999980926513671875 74.0 25.0 0
38.09999847412109375 -4.40000009536743164062 74.0 25.0 0
37.799999237060546875 -4.69999980926513671875 74.0 25.0 0
37.799999237060546875 -4.5 74.0 25.0 0
37.90000152587890625 -4.90000009536743164062 74.0 25.0 0
37.799999237060546875 -4.40000009536743164062 74.0 25.0 0
37.90000152587890625 -4.90000009536743164062 74.0 25.0 0
37.799999237060546875 -4.90000009536743164062 74.0 25.0 0
38.09999847412109375 -4.80000019073486328125 74.0 25.0 0
37.90000152587890625 -4.09999990463256835938 74.0 25.0 0
38.09999847412109375 -4.59999990463256835938 74.0 25.0 0
37.90000152587890625 -4.40000009536743164062 74.0 25.0 0
37.90000152587890625 -5.09999990463256835938 74.0 25.0 0
37.90000152587890625 -4.40000009536743164062 74.0 25.0 0
37.90000152587890625 -4.30000019073486328125 74.0 25.0 0
37.90000152587890625 -4.80000019073486328125 74.0 25.0 0
37.799999237060546875 -4.69999980926513671875 74.0 25.0 0
38.09999847412109375 -4.90000009536743164062 74.0 25.0 0
38.0 -4.90000009536743164062 74.0 25.0 0
38.0 -5.19999980926513671875 74.0 25.0 0
38.0 -5.09999990463256835938 74.0 25.0 0
37.90000152587890625 -4.80000019073486328125 74.0 25.0 0
38.09999847412109375 -4.69999980926513671875 74.0 25.0 0
38.0 -5.09999990463256835938 74.0 25.0 0
37.799999237060546875 -5.0 74.0 25.0 0
37.799999237060546875 -5.0 74.0 25.0 0
37.799999237060546875 -4.69999980926513671875 74.0 25.0 0
38.09999847412109375 -4.19999980926513671875 74.0 25.0 0
37.700000762939453125 -4.30000019073486328125 74.0 25.0 0
37.799999237060546875 -5.0 74.0 25.0 0
38.0 -4.69999980926513671875 74.0 25.0 0
37.799999237060546875 -4.90000009536743164062 74.0 25.0 0
37.90000152587890625 -4.30000019073486328125 74.0 25.0 0
37.59999847412109375 -5.69999980926513671875 74.0 25.0 0
37.5 -5.30000019073486328125 74.0 25.0 0
37.90000152587890625 -6.90000009536743164062 74.0 25.0 0
37.40000152587890625 -10.1999998092651367188 74.0 25.0 0
37.90000152587890625 -5.80000019073486328125 74.0 25.0 0
37.299999237060546875 -8.0 74.0 25.0 0
37.799999237060546875 -5.0 74.0 25.0 0
37.799999237060546875 -6.59999990463256835938 74.0 25.0 0
37.200000762939453125 -9.30000019073486328125 74.0 25.0 0
37.700000762939453125 -6.19999980926513671875 74.0 25.0 0
37.799999237060546875 -7.90000009536743164062 74.0 25.0 0
37.700000762939453125 -5.19999980926513671875 74.0 25.0 0
37.299999237060546875 -9.1000003814697265625 74.0 25.0 0
37.5 -5.09999990463256835938 74.0 25.0 0
37.200000762939453125 -5.09999990463256835938 73.0 25.0 0
37.09999847412109375 -9.5 73.0 25.0 0
37.40000152587890625 -6.59999990463256835938 73.0 25.0 0
37.700000762939453125 -5.69999980926513671875 73.0 25.0 0
37.5 -6.5 73.0 25.0 0
37.200000762939453125 -8.0 73.0 25.0 0
37.700000762939453125 -5.19999980926513671875 73.0 25.0 0
37.40000152587890625 -6.0 73.0 25.0 0
37.5 -6.40000009536743164062 73.0 25.0 0
37.59999847412109375 -6.80000019073486328125 73.0 25.0 0
37.5 -9.19999980926513671875 73.0 25.0 0
37.5 -6.30000019073486328125 73.0 25.0 0
37.799999237060546875 -5.59999990463256835938 73.0 25.0 0
37.700000762939453125 -6.40000009536743164062 73.0 25.0 0
37.59999847412109375 -5.30000019073486328125 73.0 25.0 0
37.799999237060546875 -5.59999990463256835938 73.0 25.0 0
37.5 -5.5 73.0 25.0 0
37.5 -6.09999990463256835938 73.0 25.0 0
37.90000152587890625 -5.09999990463256835938 73.0 25.0 0
37.59999847412109375 -5.90000009536743164062 73.0 25.0 0
37.5 -6.5 73.0 25.0 0
37.09999847412109375 -9.0 73.0 25.0 0
37.299999237060546875 -9.30000019073486328125 73.0 25.0 0
37.5 -4.80000019073486328125 73.0 25.0 0
37.5 -4.80000019073486328125 73.0 25.0 0
37.5 -4.80000019073486328125 73.0 25.0 0
37.5 -4.80000019073486328125 73.0 25.0 0
37.5 -4.80000019073486328125 73.0 25.0 0

GiulioRomualdi commented 1 year ago

We tried again and only the left knee went in overcurrent (maybe related)

MrAndrea commented 1 year ago

The firmware of the BAT must be updated, in fact now it is rev. 1.0.0 and it has old current limits.

The firmware must be rev. 1.2.0, for reference see:

cc @sgiraz @MSECode

MrAndrea commented 1 year ago

Ciao @sgiraz we have to remove the cover to program the BAT and check if the BAT is the rev. D (iitcode 10050).

For reference on rev. D see: https://github.com/icub-tech-iit/electronics-boards/issues/244

AntonioConsilvio commented 1 year ago

Hi @GiulioRomualdi @S-Dafarra @lrapetti, looking at the bookings, the robot is not available for this and next week, so I need the availability of the robot to perform this and other interventions.

cc @maggia80 @sgiraz @MrAndrea

DanielePucci commented 1 year ago

@AntonioConsilvio Tomorrow morning the robot is free for maintenance

MrAndrea commented 1 year ago

The BAT board was not rev. D but now it is ok: @AntonioConsilvio aligned it and @MSECode updated the firmware to the Rev. 1.2.0.

mebbaid commented 1 year ago

This issue happened again today while we are walking with the robot

S-Dafarra commented 1 year ago

This issue happened again today while we are walking with the robot

@maggia80 @MrAndrea @MSECode should we expect some message printed in the display?

MSECode commented 1 year ago

This issue happened again today while we are walking with the robot

Oks, a couple of questions for diagnostic further the problem:

were you powering the robot with the power supply or with the batteries only
do you have the prints of the yarp port of the device battery-nws-yarp, which is streaming the BMS data so we can check the last values of the current
did you stream out or saved the values of the info of the joints with matlab

Moreover, if you want I can give you a new software for the display only (therefore you don't need to update neither the icub-main nor any boards ) that allows you to view the current BAT status and the last voltage and current before the FAULT so you can check those data just before the robot is faulting and because of the display remains ON you should be able to see the last current value. I can just give you the microSD with the display software. Let me know

MSECode commented 1 year ago

This issue happened again today while we are walking with the robot

@maggia80 @MrAndrea @MSECode should we expect some message printed in the display?

I can give you the microSD so you do not have to update the EMS boards and icub-main and icub_firmware_shared, that I do not know the current status. So you can also give us feedback

MSECode commented 11 months ago

Wrapping up the issue. At the end, after several tests and using the ULink Pro Debugger, we understood that the problem of the overcurrent was due to the use of wrong resistors in the BAT board. Basically, the resistors soldered on the revision D of the board, which were necessary to raise the hw overcurrent limit from ~21A to 36A were correct in Ohm value but their shape was too big to correctly fit on the board. Therefore, it was like not having them mounted causing the overcurrent limit to remain basically the same at around 21A. After checking with the debugger that the overcurrent fault was triggered by the HSM and precisely, by an hw interrupt on the board, we checked the resistors and correctly changed them. It seems that from that moment we are not getting any overcurrent errors and also during the demo the robot was fine. Anyways, these problem should be taken under analysis since the hip joints have some spikes of current consumptions while walking but with the current limits they should work fine. On the fw side, we are releasing an update of the BAT fw that can give to the user, through a couple of boolean flags embedded on the status value, a precise indication of the cause of the fault (hardware or software and if due to oc on the hsm or on the boards themselves).

As things now, we shouldn't have anymore overcurrent faults, or at least not at the same frequency they happened lately. However, as mentioned, let's keep this behavior monitored. I would probably close this issue and eventually reopen it if we are getting faults again or define a new issue as a placeholder for this kind of problem (citing this one to have already some idea to start analyzing the problem)

cc: @maggia80 @sgiraz @GiulioRomualdi

sgiraz commented 11 months ago

Nice analysis @MSECode!

I'll leave this issue in Review for a few days, waiting for further feedback. Then, if everything looks good I'll proceed to close it.

cc @S-Dafarra @GiulioRomualdi

sgiraz commented 11 months ago

Hi guys, I guess that the problem is solved. So, I proceed to close the issue.

Feel free to reopen if necessary.

robotology / icub-tech-support