ergoCub 1.1 S/N:001 – Wrist go into Hardware fault with unknown error after few minutes of operations

vigisushrutha23 commented 6 months ago

Robot Name 🤖

ergoCub 1.1 S/N:001

Request/Failure description

After the firmware update applied on the wrists AMC, we tested our bimanual manipulation module after enabling the wrists in the calibration config. Hardware faults without any reasons were observed for the wrists in the yarpmotorgui. Screenshot from 2023-12-15 16-19-24

Detailed context

After a few starts we were able to do some actions without the wrist going into hardware fault. Once we completed the operation( duration 5min) with no issues the wrists went into hardware fault again. We were not able to put the wrists in idle and had to restart the whole robot to try for a normal full operation again.

Many starts the wrists enter hardware fault immediately and to do a normal operation it needed three restarts of the main yarprobotinterface. Screenshot from 2023-12-15 16-38-16

Additional context

No response

How does it affect you?

No response

valegagge commented 6 months ago

Hi @vigisushrutha23, have you the log of the yarprobotinterface? It can provide more information regarding the fault. Thank you!

simeonedussoni commented 6 months ago

Hi @valegagge, here we have a logging of the robotinterface and the motorgui for checking the origin of the problem.

LogError.zip

Now I try to run the bench setup in order to reproduce the error in our controlled environment

valegagge commented 6 months ago

Hi @simeonedussoni ,

checking the code I understand, that the joint is set in HW FAULT state but there isn't an error related to the motor, such as q_enc readings error, overcurrent, etc. For this in the motor GUI we have No fault detected

In detail: the yaropmotorgui gets the control mode for each joint and then, for each joint in HW FAULT asks the error type to the motor by getLastJointFault. This calls the corresponding device API getLastJointFaultRaw.

Therefore now we need to understand why the joint is in HW fault without any reason of its motor.

Stay tuned.

simeonedussoni commented 6 months ago

Recap

the error can not be reproduced in the bench setup
the error is not rtiggered w/ other demos that leave the joint in position mode
in the wbd demo the joints are driven in position direct mode
- the postition direct mode could be linked to the origin of the error
a high level application shall not rise the Fault flag
- all Faults are generated at the motor level following a diagnostic failure e.g. overcurrent, encoders out of range etc.

Actions

I am reproducing the position direct mode in the bench setup
@valegagge and @ale-git are investigating at what point in the code the error is generated

valegagge commented 6 months ago

Alongside with @ale-git, this morning we noticed some incongrouos behavior related to the management of the control_mode.

we noticed that the control mode of the joint is taken by the control mode of the entire joint-set instead of the single joint. This implies that if one joint of the joint-set (composed by the coupled joints) is in hw fault all of the joints are in hw_fault instead of only the joint with the fault is in hw fault and the other joints are in idle.
we suppose that if there is a fault in the abs-encoder, it is not notified by yarpmotorgui. (We still need to verify it)
here we don't take care of the coupled joint.

In addition, related to this ticket we saw that the motor position of the pitch joint has a strange value while the joint position is perfectly 0. Why???

We had a small catch-up meeting with @simeonedussoni, and all together we decided the following action points:

be sure that the AMC board has been configurated correctly. (Option bytes)
redo the same actions performed by @vigisushrutha23 to get more detail about the fault
have a clear idea of how the motor position is collected
check the previous point 2.

simeonedussoni commented 6 months ago

dear, yesterday together with @vigisushrutha23 we succeded in running the bimanual scenario for a quite long time, without any fault. we also tried to restart the yarprobotinterface and so on, trying to trigger this fault but failing.

the main difference with the previous run was that we launched all the components on the ergocub-torso machine rather than on the laptop, due to some network issues.

Action

we kindly ask user to perform the following actions:

launch the yarserver and yarprun
launch the yarplogger before all apps
launch the tools in ergocub-torso
in case of fault, save the yarplogger output and a screenshot of the yarpmotorgui tabs

@vigisushrutha23 where can we put this advice in order to be forwarded to all involved people?

xEnVrE commented 6 months ago

@vigisushrutha23 where can we put this advice in order to be forwarded to all involved people?

One starting point I guess might be the ergoCubSN001 Teams channel.

vigisushrutha23 commented 6 months ago

@vigisushrutha23 where can we put this advice in order to be forwarded to all involved people?

One starting point I guess might be the ergoCubSN001 Teams channel.

Yes. The teams channel is probably the best place. We can also post a note with these steps on the ergocubSn001 laptop next year for anyone using manipulation to record the data in case the fault occurs.

sgiraz commented 5 months ago

The demo went well! ✅ As a double check, tomorrow @simeonedussoni is going to test the latest binaries from https://github.com/robotology/icub-firmware/tree/devel on the wrist-setup, then we'll close this issue.

Thanks to all for your effort! 🙏🏻

cc @valegagge @simeonedussoni @valegagge @marcoaccame @maggia80 @AntonioConsilvio @vigisushrutha23

GiulioRomualdi commented 5 months ago

Yesterday during a Demo the wrist stopped moving. Even if the motor was not controllable the motorgui indicated that the joints were in position direct. Moving back the joints in position did not solve the problem.

Please notice that we use to control the wrist in positiondirect.

simeonedussoni commented 5 months ago

hi @GiulioRomualdi

since this is a different issue (and hopefully fixed with the rebase of the boards to the official release of firmware), I suggest you to open a new ticket when this condition reappears.

sgiraz commented 5 months ago

Closing in favor of https://github.com/robotology/icub-tech-support/issues/1720 where we'll track the issue mentioned in https://github.com/robotology/icub-tech-support/issues/1709#issuecomment-1900113622

cc @simeonedussoni @GiulioRomualdi

robotology / icub-tech-support