Right wrist pitch and yaw not working

[gsaponaro]

After starting robotInterface, our robot calibrates then the right wrist suddenly "falls" and we observe the following in the robot GUI:

• joint 5 (r_wrist_pitch) is IDLE
• joint 6 (r_wrist_yaw) is in HARDWARE_FAULT

In the GUI, we are then able to control these joints. However, for j6, we are able to manually set the position from +25 down to -4... and then it goes into hardware fault again.

I acquired a robotInterface log, but it is quite big to attach due to many errors like this one: [ERROR]canBusSkin error code: canDeviceNum: 8 board: 14 sensor: 13 error: 01RR.0000.0000.0000.

Please let me know if you need the full log. Here are some parts that I think are relevant:

[INFO]**********************************
[INFO]iCubInterface CAN protocol: 1.2
[INFO]Firmware report:
[INFO]sharedcan [2] joint: 0 can_address:  1 board type: 4 (BLL) version: 1.50 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 1 can_address:  1 board type: 4 (BLL) version: 1.50 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 2 can_address:  2 board type: 4 (BLL) version: 1.57 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 3 can_address:  2 board type: 4 (BLL) version: 1.57 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 4 can_address:  3 board type: 3 (4DC) version: 1.19 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 5 can_address:  3 board type: 3 (4DC) version: 1.19 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 6 can_address:  4 board type: 3 (4DC) version: 1.19 build: 80 CAN_protocol:1.2
[INFO]sharedcan [2] joint: 7 can_address:  4 board type: 3 (4DC) version: 1.19 build: 80 CAN_protocol:1.2
[INFO]**********************************
...
[ERROR][ERRORRight_Arm_Calibrator : detected an hardware fault on joint  6  before starting the calibration procedure! [DEBUG]left_hand_calibrator starting calibration of device left_hand_mc 
[ERROR]Unable to set the control mode of a joint ( j:6) in HW_FAULT
[ERROR]sharedcan [2] board 4 FAULT OVERLOAD AXIS 1
[ERROR]sharedcan [2] board 4 FAULT OVERLOAD AXIS 0

/cc @lorejam

[randaz81]

• Is this issue happening always or just sometimes?
• "detected an hardware fault on joint 6 before starting the calibration procedure!" means that the joint was already in hardware fault before starting robotInterface, probably because the hardware fault occurred during a previous calibration. Please note that by default, robotInterface does not clear previous hardware fault during the startup. In this case you have three possibilities:
  • turn off/on the motor boards to reset the board status.
  • open the robotmotorgui. click on the idle button to reset the hardware fault. Close robotInterface and start it again.
  • set 'clearHwFaultBeforeCalibration' to true in the configuration of the calibrators.
• This issue is often due to an hardware problem. Check if the wrist tendons are not loose. Check if the pin (the joint hardware limit) is not bent or damaged.

[gsaponaro]

Hi, thanks for the clarifications. The issue is happening all the time.

After making many restarts with @lorejam we observed the following:

When the hand calibrates, it calibrates first j5 then j6. After j5 calibration, j5 is at the end of the motion range, at that time it tries to move j6 to calibrate it but it goes to hardware fault because some mechanical parts are hitting each other (this can be seen also by moving the joint by hand when it is in idle).

This issue would not happen if, for example, after calibration of j5 (before moving j6), the j5 joint would be moved a bit away from the limit. Is it possible do change the position of j5 before calibrating j6?

[randaz81]

No, this cannot be changed (the two motors are coupled and sequence is the same for all robots with V1 mechanics). I still suspect an hardware issue, so I suggest to schedule a video call to check the status of the joints. @julijenv

[randaz81]

The issue is happening all the time. But when the issue started? Before it was working ok, isn't it? From the initial post, I see that you are using fw build 80, which is not the latest version. However I'd not change this, if you confirm me that your right wrist was working ok in the past. Additionally, this problem does not occur on the left wrist...

[lorejam]

Marco, it is indeed an hardware (mechanical) problem, that is present only on the right wrist. I guess the two hands are mounted slightly differently...

We saw this problem already in the past actually (with Vadim as well), probably since the iCub came back from the last update (skin).

We can send you a short vídeo later today, showing the problem.

Ciao! Lorenzo

On Thu, May 7, 2015 at 10:12 AM, randaz81 notifications@github.com wrote:

The issue is happening all the time. But when the issue started? Before it was working ok, isn't it? From the initial post, I see that you are using fw build 80, which is not the latest version. However I'd not change this, if you confirm me that your right wrist was working ok in the past. Additionally, this problem does not occur on the left wrist...

— Reply to this email directly or view it on GitHub https://github.com/robotology/icub-support/issues/23#issuecomment-99787048 .

[lorejam]

Dear all, here is the video showing the problem on the right wrist. I am moving the robot while in IDLE.

Any idea about what should we do?

Cheers, Lorenzo iCub_rightWristProblem_08052015.mp4 https://docs.google.com/file/d/0B2Ws2AH1RBDlSmpEeU91OUtaQjg/edit?usp=drivesdk Em 07/05/2015 12:29, "Lorenzo Jamone" lorenzojamone@gmail.com escreveu:

Marco, it is indeed an hardware (mechanical) problem, that is present only on the right wrist. I guess the two hands are mounted slightly differently...

We saw this problem already in the past actually (with Vadim as well), probably since the iCub came back from the last update (skin).

We can send you a short vídeo later today, showing the problem.

Ciao! Lorenzo

On Thu, May 7, 2015 at 10:12 AM, randaz81 notifications@github.com wrote:

The issue is happening all the time. But when the issue started? Before it was working ok, isn't it? From the initial post, I see that you are using fw build 80, which is not the latest version. However I'd not change this, if you confirm me that your right wrist was working ok in the past. Additionally, this problem does not occur on the left wrist...

— Reply to this email directly or view it on GitHub https://github.com/robotology/icub-support/issues/23#issuecomment-99787048 .

[julijenv]

Hi iCubLisbona01 users, as was saying randaz81 on the date of 05/05/2015, you may have to check if the end of movement pin isnt bent. Please take a look at thefollowing picture, the encircled pin: try to see if perfectly perpendicaulare to the base in which it is getting out:

Let us know

[lorejam]

Dear Julien, Randaz (and all), I attached another video in which I show the issue more in detail.

The pin looks perfectly perpendicular. It also seems to be in the same position as the other wrist (the left one, that does not show any problem). However, the metal part close to the pin looks damaged, because of the right​​ wrist colliding during the movement (as shown in the video).

​ iCubLisboa01_rightWristProblem_110515.mp4 https://docs.google.com/file/d/0B2Ws2AH1RBDlbWExRjFobld0eTA/edit?usp=drive_web ​

Could it be that the hand is not mounted properly/precisely? Or it depends on how the tendons are tighten?

Let us know, thanks for the support. Lorenzo

On Mon, May 11, 2015 at 10:49 AM, julien notifications@github.com wrote:

Hi iCubLisbona01 users, as was saying randaz81 on the date of 05/05/2015, you may have to check if the end of movement pin isnt bent. Please take a look at thefollowing picture, the encircled pin: try to see if perfectly perpendicaulare to the base in which it is getting out: [image: pin_end_movment] https://cloud.githubusercontent.com/assets/7140395/7562286/9dc47210-f7d3-11e4-8668-df9721b1539c.JPG

Let us know

— Reply to this email directly or view it on GitHub https://github.com/robotology/icub-support/issues/23#issuecomment-100838070 .

[gsaponaro]

Hi all, do you have any other suggestion on this issue, considering the latest information provided by @lorejam on 11 May? Thanks.

[gsaponaro]

/cc @julijenv @super-ste

[julijenv]

Hi @gsaponaro and @lorejam, It is a kind of known issue of the wrist V1, the right wrist block after a certain period of use, you will have to file a bit the metallic bit holding one of the cable at the base of the wrist. see pictuer on our red icub, the last robot with V1 wrist.

Just file a bit in order to make jumped the wrist, try to get a 45 degree angle at the base of the metallic plate. Let us know

[gsaponaro]

Thanks very much, @julijenv . We will proceed with your suggestion after the holidays. Have a good August! In the meantime, today we just inspected the wrist further, and it seems we identified these problems:

1)

The metal bit that needs to be filed, as you suggest (will do that asap).

2)

The steel cable of r_hand_finger (hand finger adduction/abduction) was broken, and probably the corresponding Hall effect sensor too: when we manually move this joint, the encoder reading in the motor GUI does not change, it stays fixed at the value of 447 degrees.

3)

Two little screws were missing on a metal panel of the right hand. Likely, it has been like this for a while but only now did we notice. This arm was at IIT for repairs during January 2014, and the whole robot again during summer 2014. We haven't really touched the hand since then, in particular not those screws or holes. Probably they had not been put back.

Ciao

[julijenv]

Hi iCubLisboa01 users (@lorejam e @gsaponaro), So as mentionned on your last comment, you identified quite a lot of issue on the same hand. As a matter of fact 3 things need to be done: -File the metal plate -Rewire the abduction (http://wiki.icub.org/wiki/Tendons_and_replacement) and download the tendonsHand.pdf and check thumb_oppose description -Hall sensor not working The main question that rise in my mind when i am writing is : are you able to fix everything by yourself if we give you support via skype ( have you all the replacement pieces to try to put up and runngin the hand - I mean cable 0.63mm non coated? for the thumb abduction) or do you think that you wanna send the forearm to us during that you are in holidays? Let us know

[lorejam]

Hi @julijenv https://github.com/julijenv, we made a number of tests here in Lisbon.

In relation to your previous email: 1) we filed the metal plate and bent the metal pin, and now the wrist can move with no problem ;) 2) we rewired the abduction correctly 3) we made a number of tests for the Hall sensor of the abduction (it is the one above the index finger, right? are there any other sensors measuring the abduction?).

However: 1) the wrist can now move (if it is in idle, and I move it myself), but one of the motor actuating the wrist is not working (the motor highlighted in the attached image). So, if I try to move the wrist from yarpMotorGui, I can move joint 5 (but the motion is not the desired one) and I cannot move joint 6 at all. 2) the abduction motor seems not to work as well (I cannot move it from yarpMotorGui). 3) the Hall sensor always displays -447 in the yarpMotorGui. We took the sensor out from its location, and put it close to another magnet, but still it displays -447 on the yarpMotorGui. However, if we check the voltage measured on the board, the voltage changes if we put the magnet close to the sensor... So, the sensor is actually working, the sensor measurements arrives to the board (no wiring problem), but then it is not displayed on the yarpMotorGui, nor used for control reference (apparently). 4) Also, joint 4 always starts in IDLE after running the robotInterface, but then if I manually put "position" from the yarpMotorGui, then it works normally (so, this is a minor issue, I would say... but still an issue).

Any idea? Is there any way we can test the motor directly bypassing yarpmotorgui? I was trying "set vmo" from yarp rpc but it seems not to work anymore... ? Any other solution (but directly plugging a voltage to the motor input?) ?

Thanks a lot! Best, Lorenzo, Giovanni, Ricardo

On Mon, Aug 3, 2015 at 11:05 AM, julien notifications@github.com wrote:

Hi iCubLisboa01 users (@lorejam https://github.com/lorejam e @gsaponaro https://github.com/gsaponaro), So as mentionned on your last comment, you identified quite a lot of issue on the same hand. As a matter of fact 3 things need to be done: -File the metal plate -Rewire the abduction (http://wiki.icub.org/wiki/Tendons_and_replacement) and download the tendonsHand.pdf and check thumb_oppose description -Hall sensor not working The main question that rise in my mind when i am writing is : are you able to fix everything by yourself if we give you support via skype ( have you all the replacement pieces to try to put up and runngin the hand - I mean cable 0.63mm non coated? for the thumb abduction) or do you think that you wanna send the forearm to us during that you are in holidays? Let us know

— Reply to this email directly or view it on GitHub https://github.com/robotology/icub-support/issues/23#issuecomment-127182942 .

Lorenzo Jamone - Research Associate http://lorejam.blogspot.pt/ ljamone@isr.tecnico.ulisboa.pt Tecnicojasv@isr.tecnico.ulisboa.pt

VisLab - http://vislab.isr.ist.utl.pt Instituto Superior Técnico - http://tecnico.ulisboa.pt/

Institute for Systems and Robotics, Torre Norte (7th floor). Av. Rovisco Pais, 1 1049-001 Lisbon, Portugal

Office (room 7.25): +351218418299 Laboratory (room 7.26): +351218418050

[lorejam]

Hi @julijenv https://github.com/julijenv, please, disregard my previous email. Consider this one instead.

We made a number of tests here in Lisbon.

In relation to your previous email: 1) we filed the metal plate and bent the metal pin, and now the wrist can move with no problem ;) 2) we rewired the abduction correctly 3) we made a number of tests for the Hall sensor of the abduction (it is the one above the index finger, right? are there any other sensors measuring the abduction?).

However: 1) the wrist can now move (if it is in idle, and I move it myself), but one of the motor actuating the wrist is not moving (the motor highlighted in the attached image). So, if I try to move the wrist from yarpMotorGui, I can move joint 5 (but the motion is not the desired one) and I cannot move joint 6 at all. I did not try to directly apply a voltage to the motor, because I don't know how to access it. 2) the abduction motor works (we moved it by directly applying a voltage), but I cannot move it from the yarpMotorGui. 3) the Hall sensor always displays -447 in the yarpMotorGui. We took the sensor out from its location, and put it close to another magnet, but still it displays -447 on the yarpMotorGui. However, if we check the voltage measured on the board, the voltage changes if we put the magnet close to the sensor... So, the sensor is actually working, the sensor measurements arrives to the board (no wiring problem), but then it is not displayed on the yarpMotorGui, nor used for control reference (apparently). 4) Also, joint 4 always starts in IDLE after running the robotInterface, but then if I manually put "position" from the yarpMotorGui, then it works normally (so, this is a minor issue, I would say... but still an issue).

Any idea? Is there any way we can test the motor directly bypassing yarpmotorgui? I was trying "set vmo" from yarp rpc but it seems not to work anymore... ? Any other solution (but directly plugging a voltage to the motor input?) ?

Thanks a lot! Best, Lorenzo, Giovanni, Ricardo

On Tue, Sep 8, 2015 at 12:47 PM, Lorenzo Jamone lorenzojamone@gmail.com wrote:

Hi @julijenv https://github.com/julijenv, we made a number of tests here in Lisbon.

In relation to your previous email: 1) we filed the metal plate and bent the metal pin, and now the wrist can move with no problem ;) 2) we rewired the abduction correctly 3) we made a number of tests for the Hall sensor of the abduction (it is the one above the index finger, right? are there any other sensors measuring the abduction?).

However: 1) the wrist can now move (if it is in idle, and I move it myself), but one of the motor actuating the wrist is not working (the motor highlighted in the attached image). So, if I try to move the wrist from yarpMotorGui, I can move joint 5 (but the motion is not the desired one) and I cannot move joint 6 at all. 2) the abduction motor seems not to work as well (I cannot move it from yarpMotorGui). 3) the Hall sensor always displays -447 in the yarpMotorGui. We took the sensor out from its location, and put it close to another magnet, but still it displays -447 on the yarpMotorGui. However, if we check the voltage measured on the board, the voltage changes if we put the magnet close to the sensor... So, the sensor is actually working, the sensor measurements arrives to the board (no wiring problem), but then it is not displayed on the yarpMotorGui, nor used for control reference (apparently). 4) Also, joint 4 always starts in IDLE after running the robotInterface, but then if I manually put "position" from the yarpMotorGui, then it works normally (so, this is a minor issue, I would say... but still an issue).

Any idea? Is there any way we can test the motor directly bypassing yarpmotorgui? I was trying "set vmo" from yarp rpc but it seems not to work anymore... ? Any other solution (but directly plugging a voltage to the motor input?) ?

Thanks a lot! Best, Lorenzo, Giovanni, Ricardo

On Mon, Aug 3, 2015 at 11:05 AM, julien notifications@github.com wrote:

Hi iCubLisboa01 users (@lorejam https://github.com/lorejam e @gsaponaro https://github.com/gsaponaro), So as mentionned on your last comment, you identified quite a lot of issue on the same hand. As a matter of fact 3 things need to be done: -File the metal plate -Rewire the abduction (http://wiki.icub.org/wiki/Tendons_and_replacement) and download the tendonsHand.pdf and check thumb_oppose description -Hall sensor not working The main question that rise in my mind when i am writing is : are you able to fix everything by yourself if we give you support via skype ( have you all the replacement pieces to try to put up and runngin the hand - I mean cable 0.63mm non coated? for the thumb abduction) or do you think that you wanna send the forearm to us during that you are in holidays? Let us know

— Reply to this email directly or view it on GitHub https://github.com/robotology/icub-support/issues/23#issuecomment-127182942 .

Lorenzo Jamone - Research Associate http://lorejam.blogspot.pt/ ljamone@isr.tecnico.ulisboa.pt Tecnicojasv@isr.tecnico.ulisboa.pt

VisLab - http://vislab.isr.ist.utl.pt Instituto Superior Técnico - http://tecnico.ulisboa.pt/

Institute for Systems and Robotics, Torre Norte (7th floor). Av. Rovisco Pais, 1 1049-001 Lisbon, Portugal

Office (room 7.25): +351218418299 Laboratory (room 7.26): +351218418050

Lorenzo Jamone - Research Associate http://lorejam.blogspot.pt/ ljamone@isr.tecnico.ulisboa.pt Tecnicojasv@isr.tecnico.ulisboa.pt

VisLab - http://vislab.isr.ist.utl.pt Instituto Superior Técnico - http://tecnico.ulisboa.pt/

Institute for Systems and Robotics, Torre Norte (7th floor). Av. Rovisco Pais, 1 1049-001 Lisbon, Portugal

Office (room 7.25): +351218418299 Laboratory (room 7.26): +351218418050

[julijenv]

Hi @lorejam, @gsaponaro,

1) the wrist can now move (if it is in idle, and I move it myself), but one of the motor actuating the wrist is not working (the motor highlighted in the attached image).

missing attach :P I guess it is joint 6 Did you try to directly power it?

3) we made a number of tests for the Hall sensor of the abduction (it is the one above the index finger, right? are there any other sensors measuring the abduction?).

I dont know which abduction joint you are talking about??? thumb abduction or fingers abduction? for the fingers abduction the sensor is located above the pinky finger meanwhile the one from the thumb abduction is located inside the palm beneath the tumb - see pictures below:

4) Also, joint 4 always starts in IDLE after running the robotInterface, but then if I manually put "position" from the yarpMotorGui, then it works normally (so, this is a minor issue, I would say... but still an issue).

can we get a log file to see the error message description of this behaviour?

Anyway; can we get in touch via skype i think it would be better to understand what u can do, what u did and what's goin on :P

skype ID : icub.support

C U around

[lorejam]

Ok, thanks @julijenv https://github.com/julijenv,

1) so here is the wrist motor that is not working (and yes, I think it is joint 6, even if the joints are coupled actually...):

I don't know how to directly power the motor... where exactly should I apply the voltage? On the motor (the case is closed...) or on the board? ​ 3) I am talking about fingers abduction and yes, the Hall sensor is the one above the pinky finger.

4) I will send you the log ASAP...

On Tue, Sep 8, 2015 at 3:40 PM, julien notifications@github.com wrote:

Hi @lorejam https://github.com/lorejam, @gsaponaro https://github.com/gsaponaro gsaponaro,

1) the wrist can now move (if it is in idle, and I move it myself), but one of the motor actuating the wrist is not working (the motor highlighted in the attached image).

missing attach :P I guess it is joint 6 Did you try to power it ?

3) we made a number of tests for the Hall sensor of the abduction (it is the one above the index finger, right? are there any other sensors measuring the abduction?).

I dont know which abduction joint you are talking about??? thumb abduction or fingers abduction? for the fingers abduction the sensor is located above the pinky finger meanwhile the one from the thumb abduction is located inside the palm beneath the tumb - see pictures below: [image: thumbsensor1] https://cloud.githubusercontent.com/assets/7140395/9737184/a6833e6a-5645-11e5-940d-c04fe16b561e.jpg [image: thumbsensor2] https://cloud.githubusercontent.com/assets/7140395/9737183/a67a1434-5645-11e5-8c2b-ccc2636056f5.jpg

4) Also, joint 4 always starts in IDLE after running the robotInterface, but then if I manually put "position" from the yarpMotorGui, then it works normally (so, this is a minor issue, I would say... but still an issue).

can we get a log file to see the error message description of this behaviour?

Anyway; can we get in touch via skype i think it would be better to understand what u can do, what u did and what's goin on :P

skype ID : icub.support

C U around

— Reply to this email directly or view it on GitHub https://github.com/robotology/icub-support/issues/23#issuecomment-138586830 .

Lorenzo Jamone - Research Associate http://lorejam.blogspot.pt/ ljamone@isr.tecnico.ulisboa.pt Tecnicojasv@isr.tecnico.ulisboa.pt

VisLab - http://vislab.isr.ist.utl.pt Instituto Superior Técnico - http://tecnico.ulisboa.pt/

Institute for Systems and Robotics, Torre Norte (7th floor). Av. Rovisco Pais, 1 1049-001 Lisbon, Portugal

Office (room 7.25): +351218418299 Laboratory (room 7.26): +351218418050

[gsaponaro]

Regarding the robotInterface log, I recently uploaded one here: https://github.com/robotology/icub-support/issues/43 - file is rather large because it's full of skin warnings. However, information about the joints in question should probably be there.

[julijenv]

Hi @gsaponaro and @lorejam , From the logfile it is so full of error message coming from the skinboard that it is quite unclear qht's going wrong could you please remove the skin parts from the icub_all.xml file in order to view just the interesting problem with the motor of the hands. Would it be possible to get in touch via skype in order to explain you how to test the motors? let me know

[gsaponaro]

Logfile: I cleaned it (with grep -v "canBusSkin error code" iCubLisboa01_robotInterface_150729.txt > iCubLisboa01_robotInterface_150729_noskin.txt) and uploaded it here: https://gist.github.com/gsaponaro/4114da4cfe141e68bf2c

Skype: we'll contact you asap, thanks.

[gsaponaro]

Thanks @julijenv, for now all problem with the right arm are solved! After having swapped the fingers 4DC and the wrist 4DC boards. :)

However, one of the 4DC boards is probably unreliable (the one that was responsible for all the problems in the previous location). If anything bad happens again, we'll try re-flashing the firmware and let you know.