search

robotology / icub-tech-support

Virtual repository that provides support requests for individual robots
GNU General Public License v2.0
20 stars 2 forks source link

Uncontrolled state spotted on joints controlled via MC4Plus #673

Closed S-Dafarra closed 2 years ago

S-Dafarra commented 6 years ago

Description of the failure

The left arm goes in a state which is not down, i.e. the motors are not completely switched off, but yet the joints starting from the wrist prono-supination cannot be controlled and are kind of "compliant".

Detailed conditions and logs of the failure

logArmDown_part1.txt logArmDown_part2.txt

Image of the arm: image I can move easily the prono-supination, while the wrist is stuck to the upper limit.

cc @julijenv

S-Dafarra commented 6 years ago

Also there is a super annoying whistle coming from the hand. If we squeeze the fingers (moving the l_hand_fingers joint), it stops.

julijenv commented 5 years ago

Hi @S-Dafarra , still on going this problem?!?!

S-Dafarra commented 5 years ago

Yes, actually it happens on both arms. For example, yesterday it happened on the right arm. Interestingly we also pressed the fault, but the forearm remained in that strange state.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

S-Dafarra commented 5 years ago

This issue is still occurring.

S-Dafarra commented 5 years ago

It just happened today, again on the left arm. As far as I know, this is still happening on both arms. Here a more recent log log_icub-head_unresponsiveWrist.txt

After a quick F2F talk with @julijenv we suspect this issue may be related to a board resetting, maybe after a voltage drop, which causes the board to remain in an uncontrolled state with a constant PWM output. Maybe @valegagge or @marcoaccame may have some opinion on this.

julijenv commented 5 years ago

Hi @valegagge , if you time to check that and give me your thoughts about it I'd be happy, thx in advacnce

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

S-Dafarra commented 4 years ago

The issue is still occurring.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

S-Dafarra commented 4 years ago

It still happens.

S-Dafarra commented 4 years ago

After a quick f2f with @julijenv we understood that we need to check whether the board is still pingable when this problem is occurring. This may give us some more hints of what is going on.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

S-Dafarra commented 4 years ago

I am not going to give up on this stale-bot! :joy:

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale[bot] commented 4 years ago

This issue has been automatically closed due to inactivity. Feel free to open it again if needed.

S-Dafarra commented 4 years ago

Yet another log. BY accident in the end we switched off all the boards by mistake before stopping the yarprobotinterface. Nevertheless, the left wrist was in this status since a while.

log_icub-head_yarprobotinterface_1705_left_wrist_uncontrollable.txt

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

S-Dafarra commented 4 years ago

Still happening

pattacini commented 4 years ago

Let's try to tackle this with the help of @davidetome

davidetome commented 4 years ago

Ciao,

I took a look at the logs provided. The fault that causes this behaviour is most likely : 

 6450 1461.226705 <ERROR>   FATAL: EthResource::getRemoteValue() DID NOT have replies from BOARD left_arm-eb24-j4_7 with IP 10.0.1.24  even after 0.200232 seconds: CANNOT PROCEED ANY FURTHER

As already pointed out in this comment, does it seem that the board is resetting for some reasons (a short circuit in a certain position of the arm? a wire not well connected? A fault on the mc4plus board?)

I propose to do some tests tomorrow morning but I need help, maybe we need to uncover the arm, check for connections and maybe replacing the board in the last instance. @julijenv are you available for this?

@S-Dafarra just a couple of questions:

  1. Is the robot available tomorrow morning?
  2. Is the failure replicable in some conditions (positions) or it happens randomly?
  3. Are the firmware of the boards updated? If not, can we do that?
  4. Is the sofware running on laptop/server and on the icub-headupdated?

I look forward to your replies, thanks!

cc @pattacini

pattacini commented 4 years ago

They're doing many tests these days with Greeny. Let's see if the robot is available.

S-Dafarra commented 4 years ago

Hi @davidetome

  • Is the robot available tomorrow morning?

Probably yes. We are planning to use it today full day, but tomorrow it should be free. The robot is still reserved for the iRonCub experiments, but they got postponed of one week. Unfortunately, we weren't able to cancel the booking. I think tomorrow morning is ok.

  • Is the failure replicable in some conditions (positions) or it happens randomly?

If it is repeatable, we did not find how. It happens from time to time, especially under stress (when walking or doing other dynamic demos), even without moving the arms. Once, during a demo, it was happening by simply waving the hand. In general, it seems to happen only when moving. Unfortunately, it seems pretty random.

  • Are the firmware of the boards updated? If not, can we do that?

I tried to update about two months ago, but if you need to update the boards, feel free to do it. If problems occur, we would need to fix them anyway :wink:

4. Is the sofware running on laptop/server and on the icub-headupdated?

Right now we are using the latest release on the icub-head. The configuration files are over master. Anyways, I remember this issue happening since the first time we started the robot after being assembled.

Even if we are not spotting the actual issue, do you think it may be possible to work it around via sfotware?

davidetome commented 4 years ago

Ciao @S-Dafarra ,

Answering your question I think it is not so easy to work it around also because it involves both FW and SW, especially if the board is resetting as it seems; I mean when yarprobotinterface starts configures all the boards and services and if a board goes down and up again it can't be anymore usable until yarprobotinterface restarts. But on this point @marcoaccame is certainly the most suitable person.

I spoke with @julijenv, and he will be at CRIS until about 12.00am. He also noticed that the robot is just felt down so is going to check if there are damages and let me know if the robot is still usable. If yes, as starting point, I can come and try to reproduce the failure checking for the EB24 board reset.

Anyway, can you please advise your colleagues and leave the robot powered off so tomorrow I can bring it to proto lab?

cc @pattacini @Fabrizio69

S-Dafarra commented 4 years ago

Answering your question I think it is not so easy to work it around also because it involves both FW and SW, especially if the board is resetting as it seems; I mean when yarprobotinterface starts configures all the boards and services and if a board goes down and up again it can't be anymore usable until yarprobotinterface restarts.

I see. We usually reboot also the boards. I may be wrong, but restarting the yarprobotinterface may not be enough. I was hoping that with some yarpmotorgui macumba it was possible to get them back (by triggering the calibration again maybe), but I am just wondering. I totally understand the complexity.

He also noticed that the robot is just felt down

Yeah, I have some responsibility :grin:

let me know if the robot is still usable.

For this I would trust the opinion of @julijenv

Anyway, can you please advise your colleagues and leave the robot powered off so tomorrow I can bring it to proto lab?

Yes, I will either do it by myself or ask someone to do it.

davidetome commented 4 years ago

Just an update, the robot is not starting due to a problem on the power supply (backpack, some mechanical parts are bent). @julijenv is disassembling the part and going to check it out.

davidetome commented 4 years ago

Another Update,

The robot now starts again (it was a problem with BMON board), @julijenv straightened the bent mechanical part. We faced another issue with the EXT FAULT that was seen pressed by yarprobotinterface tough the button was not really pressed. Davide Gandini checked the wires and cabling fixing it (probably the connector not well inserted after the fall).

Now I'm going to move the left-arm for a while using a sequence in yarpmotorgui and checking if the board EB24 restarts

davidetome commented 4 years ago

Las t update

the robot is running this sequence on the left-arm for about 2 hours and the board EB24 is still up and running.

I propose to plan another day (at least) with @julijenv help to do some tests uncovering the arm.

@S-Dafarra you can take back the robot for the moment :+1:

cc @pattacini

S-Dafarra commented 4 years ago

the robot is running this sequence on the left-arm for about 2 hours and the board EB24 is still up and running.

Thanks a lot @davidetome. Just to understand, more or less which is the motion of the arm you were trying? Unfortunately, we never understood the root cause. Sometimes it happened also when running the torque-controlled Yoga++, where the arms are moved very little. Also, it happens on both arms.

In case it happens again while doing some other experiments, can we call you or there is something we can try/save?

davidetome commented 4 years ago

Ciao @S-Dafarra , I simply cycled some position moving most of the joint of the left-arm using the sequence posted above in the yarpmotorgui to see if the board EB24 disappear. Honestly, I did not understand that this happens on both the arms, so this changes a little bit the focus. We could consider asking an opinion also to firmware guys (@marcoaccame @ale-git )...

cc @julijenv

S-Dafarra commented 4 years ago

Honestly, I did not understand that this happens on both the arms, so this changes a little bit the focus.

I probably should have changed the title. I mentioned in a few comments above it was happening on both arms. Unfortunately, I was able to get a usable log only for the left arm. Most of the time it happens after an intensive session, where the log is already saturated since a while.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

S-Dafarra commented 3 years ago

Interestingly, this happened once also on iCub3. So, I guess it is not a problem specific to iCubGenova04.

julijenv commented 3 years ago

I dont have too much time to deal with this specific issue right now. But at the moment as you define it as non machine specific I would consider it a software issue. and I will let more competent people try to figure out, if it is software related or not before investigated too much time uselessly. @pattacini

S-Dafarra commented 3 years ago

We noticed a similar behavior also for the neck joints. From time to time, the neck joints cannot be controlled anymore and we have to reboot the motors. I suspect that the issue is the same, and I think it has to do with the mc4plus boards that control all these joints.

@julijenv @pattacini let me know if you prefer me to change the title of this issue or to open a new one.

cc @DanielePucci

pattacini commented 3 years ago

let me know if you prefer me to change the title of this issue or to open a new one.

Title updated! Better off sticking to this issue to keep an eye on the long history. In this sprint, @davidetome will get in touch w/ you to see how to proceed with this.

S-Dafarra commented 3 years ago

let me know if you prefer me to change the title of this issue or to open a new one.

Title updated! Better off sticking to this issue to keep an eye on the long history. In this sprint, @davidetome will get in touch w/ you to see how to proceed with this.

Ok! Notice that we don't have a lot of time. Starting from next week we will be involved in some shootings outside the lab. It would make our life much easier if we are able to face this during this week.

At this point, I suspect that the problem is mainly related to firmware as it involves several boards on several robots. The nice thing is that we can use iCub3 for the tests even if we keep the greeny busy.

I don't know if it is related, but I remember a bug on the FT boards that caused them to get stuck because of communication issues: https://github.com/robotology/icub-firmware/issues/102

Is it possible that a similar bug is also present in the mc4plus firmware?

pattacini commented 3 years ago

We'll see how we can handle that, although it's difficult to promise to sort out a longstanding problem in a few days.

cc @davidetome @ale-git

davidetome commented 3 years ago

Ciao @S-Dafarra , if it is feasible me and @ale-git would be on this on Thursday morning(at least) to do some test.. is iCubGenova09 available?

cc @pattacini @DanielePucci

DanielePucci commented 3 years ago

I further checked with @S-Dafarra and it is ok on Thursday! Please just book the robot

CC @robotology/iit-dynamic-interaction-control

DanielePucci commented 3 years ago

CC @ale-git @davidetome @pattacini

davidetome commented 3 years ago

Interestingly, this happened once also on iCub3. So, I guess it is not a problem specific to iCubGenova04.

@S-Dafarra since we're going to use iCub3 this morning. on which joint did it happened?

cc @ale-git

pattacini commented 3 years ago

From the OP:

joints starting from the wrist prono-supination cannot be controlled and are kind of "compliant"

plus:

We noticed a similar behavior also for the neck joints. From time to time, the neck joints cannot be controlled anymore and we have to reboot the motors. I suspect that the issue is the same, and I think it has to do with the mc4plus boards that control all these joints.

davidetome commented 3 years ago

@pattacini , I've read but I was asking if on iCub3 it happened on the neck, on the wrist or on both of them

S-Dafarra commented 3 years ago

@pattacini , I've read but I was asking if on iCub3 it happened on the neck, on the wrist or on both of them

On iCub3 we noticed only the wrists for the time being.

S-Dafarra commented 3 years ago

In any case, let me stress again some of the things we noticed:

davidetome commented 3 years ago

We moved the robot for a while trying (in particular the left forearm) to replicate the fault but it didn't happen. Anyway looking again at the logs with @ale-git we agree that it should be due to an HW problem since the boards restart.

Consider that all mc4plus mounted on the robots until now are not functionally tested by the supplier.

We agreed with @S-Dafarra to try changing the mc4plus controlling the forearms (EB24 and EB27) of the iCubGenova04 and in the meanwhile check wirings both ETH and Power.

output

📝 @S-Dafarra : The code of mc4plus is 5443.D (last rev.)

cc @pattacini @DanielePucci @maggia80

DanielePucci commented 3 years ago

If the output of the analysis is to change the mc4plus, let us know when you would like to proceed with the substitution

pattacini commented 3 years ago

Hi @DanielePucci

The availability of the boards and when to do this replacement needs to be discussed internally w/ @Fabrizio69 and @julijenv first. Stay tuned.

S-Dafarra commented 3 years ago

The problem of planning this after the demo, is that the usage of the robot will drop. For iCubGenova04 in particular, the forearms might be removed for a while as well. I am just afraid that a late replacement may give us an answer on whether it worked or not with a lot of delay.