iCubNancy01 S/N:021 – multiple cables broken, left and right hands

olivierrochel-inria commented 1 year ago

Robot Name 🤖

iCubNancy01 S/N:021

Request/Failure description

After an impact on a table, several cables of the left and right hands were broken. Preliminary tests show no electronics failure, only mechanical ones. Any help on the best strategy to fix this would be warmly appreciated.

Detailed context

During an experiment in which the robot was supposed to point at things both arms hit a table. As a result at least two cables are broken or disconnected on the left hand, one on the right hand (see pictures below). Let us know if you need more views, or additional information.

What would you recommend to repair this ? And what could be the associated time/difficulty, or cost, in any case ? sending the robot (or only the arms) back to the iit, repairing it ourselves, or having someone with the right expertise to visit us and fix everything here ... ?

Left hand: left_hand

Right hand: right_hand

this doesn't look nice but possibly unrelated (left elbow):

@jbmouret @serena-ivaldi

Additional context

No response

How does it affect you?

No response

github-actions[bot] commented 1 year ago

⚠️ Detected missing mandatory information

Hi @olivierrochel-inria 👋🏻

Some of the following points need your attention.

You are required to:

[x] enter the Robot Name.
[x] give the issue a meaningful Title.
[ ] give the issue a meaningful Description (at least 100 characters).

_{Please, mark the points above as solved once done.} [^1]: To get to know the allowed options for the dropdown fields, you may peruse the template.

olivierrochel-inria commented 1 year ago

Additional pictures, left hand, forearm and hand covers removed:

sgiraz commented 1 year ago

What would you recommend to repair this ? And what could be the associated time/difficulty, or cost, in any case ? sending the robot (or only the arms) back to the iit, repairing it ourselves, or having someone with the right expertise to visit us and fix everything here ... ?

Hi @olivierrochel-inria,

Unfortunately, the forearms seem to be seriously damaged. I recommend you send us both arms in order to repair them. Below, I have provided instructions to assist you in removing the forearms.

Note Your robot was initially a v1.x model, which was upgraded to the ETH version in April 2016. I am unsure about the version of your forearms, so the following instructions are for the latest v2.7 iCub version. If you have an older version, the instructions may differ slightly. If you have any doubts or need more precise instructions, please post a picture of the inside of the upper arm covers, highlighting the support of the boards mounted on them.

🪛 How to remove the forearms from iCub 2.7

Remove the upper arm covers. This can be achieved by removing all the small screws on the skin.
Detach the CAN Molex connectors that connect the mtb4 boards to the mc4plus-eb24 and mc4plus-eb27. Here is the Logic scheme of the connections (See pages 1, 5, and 6): https://github.com/icub-tech-iit/electronics-wiring-public/blob/master/icub2.7/pdf/iCub2.7_E3.6.3_14838_Logic.pdf
Remove the 4 screws (circled in red) of the steel support with the electronics board (we need the electronics).

Remove the following 3 screws (circled in red):

Important Before shipping, make sure that the unmounted lower arms look like the following:

📦 Shipment info

IIT - Istituto Italiano di Tecnologia Via Morego 30 I-16163 Genoa, ITALY Att.n: Fabrizio Larosa (iCub Tech, CIRS) Phone: +39 010 28 98 223

IMPORTANT: please safely pack the goods using the maximum care to absorb shocks. If possible, indicate "fragile" on the parcel. As soon as you have a tracking number, pls let us know it so we may follow the shipping. If possible, ensure the goods with the shipping company - indicative (reasonable) values you may put 30keur for 1 lower-arm (forearm, wrist and hand altogether). If you need further info, pls let me know.

cc @andreaderito @Fabrizio69 @maggia80 @AntonioConsilvio @GiacomoVassallo13

olivierrochel-inria commented 1 year ago

Hi @sgiraz , all, Thank you very much for your answer. As you suggested this looks like a different version (V1), but the assembly is very similar : I will disconnect the boards (labeled 2B3, 2B4 ...) from above, remove the 3 screws from step 4, and send you the disconnected lower arms together with the electronics, as in you last picture.

upper_right_01

olivierrochel-inria commented 1 year ago

The arms have been disconnected (see below), please let us know if you think something is missing.

arms

sgiraz commented 1 year ago

The arms have been disconnected (see below), please let us know if you think something is missing.

Well done @olivierrochel-inria!

Everything looks good! 🚀 Feel free to prepare the package for the shipment of the forearms.

Before the shipment we have to estimate the costs for the maintenance intervention. Please, wait for our proposal and confirmation before proceeding with the shipment.

cc @andreaderito @maggia80

olivierrochel-inria commented 1 year ago

Thank you @sgiraz for your feedback, ok - we will wait for the quotation.

Minor addition to the damage assessment: the black cover of the left hand is broken (see picture), not a big deal compared to the broken tendons - a new one may be needed, or we should be able to fix it on our side (or maybe 3D-print a new one), feel free to suggest the best way to repair it in any case.

(covers removed for the picture, will be reassembled and shipped with the rest) covers

andreaderito commented 1 year ago

Dear @olivierrochel-inria , I'm in charge of preparing the quotation - pls be so kind as to pack the goods and communicate in this issue 1) dimensions and weight (also indicative) of the parcel 2) Exact address of the pick up of the goods 3) One or 2 reference persons at the pick-up (complete address + email address + office telephone number, possibly mobile - thx 4) If there are any traffic restrictions useful to know for express courier, just tell me. Thank you in advance - Andrea Derito

sgiraz commented 1 year ago

(covers removed for the picture, will be reassembled and shipped with the rest)

Don't worry, we'll repair/replace the covers alongside the other broken/missing parts.

cc @Fabrizio69

olivierrochel-inria commented 1 year ago

Thank you @andreaderito ,

I'm a bit confused, does that mean that you can organize transport in both directions? Or do you need this information for the return shipment ? I thought from previous exchanges that we were supposed to send the arms, and you'll take care of the return only - it's obviously simpler for us if you can manage both directions, but I'd like to be sure of the procedure to follow. Please accept my apologies if I misunderstood something.

andreaderito commented 1 year ago

Dear @olivierrochel-inria you are right, the first idea was you to manage the shipping because it was not that clear about the possible costs for the repair - but in the meantime it has been cleared and my colleagues have been able to estimate the maintenance service costs, so now I may be in condition to prepare the quotation and I may include both shipping (it simplifies things for you, and we may also activate our insurance policy on the parcel). So in brief I'll prepare the quotation and send it to you. Pls let me kindly have your answers to my questions above so I proceed with the quote, thanks and best

sgiraz commented 7 months ago

Hi @olivierrochel-inria,

The maintenance activities have been concluded successfully and the forearms are ready to be shipped back to you!

Let us know when you received the package.

cc @AntonioConsilvio @andreaderito @Fabrizio69

sgiraz commented 6 months ago

Hi @olivierrochel-inria,

Recently, you should received the forearms back. Have you had the opportunity to mount them? If yes, and everything is ok, I think we can close the issue, right?

cc @AntonioConsilvio

olivierrochel-inria commented 6 months ago

Hi, we received the arms last week, thanks. I'm just back in the lab, we plan to mount them asap (maybe today if all goes well), I'll keep you informed here.

olivierrochel-inria commented 6 months ago

Hi again, sorry for the delay.

the first try after remounting the arms failed with some CAN errors, we discovered a loose wire on a molex connector (possibly ripped off during remounting, or transport). This has been fixed, no CAN errors anymore, but we're now faced with: 4.570328 from BOARD 10.0.1.3 (right_arm-eb3-j0_3), src LOCAL, adr 0, time 77s 635m 491u: (code 0x05000016, par16 0x4008 par64 0xffffffffffff1000) -> CFG: EOtheEncodeerReader cannot be configured: verification fails. p16&0xf000: number of joint; primary encs: failure mask in p16&0x000f and errorcodes in p64&0x0000ffff; secondary encs: failure mask in p16&0x00f0 and errorcodes in p64&0xffff0000 + . 17.487967 from BOARD 10.0.1.1 (left_arm-eb1-j0_3), src LOCAL, adr 0, time 90s 548m 528u: (code 0x02000010, par16 0x0002 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + . 17.488006 from BOARD 10.0.1.3 (right_arm-eb3-j0_3), src LOCAL, adr 0, time 90s 547m 503u: (code 0x02000010, par16 0x0000 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + . ...

and many similar errors in the logs. Any help on how to fix or at least diagnose this will be welcome, I don't really know what to do next...

Full log attached log_pc104_yarprobotinterface_1457.txt

thanks, O.

AntonioConsilvio commented 6 months ago

Hi @olivierrochel-inria, from the log, it seems that the robot cannot find some CAN boards. To better understand the cause of the problem, I ask you to perform a few steps to get more information:

Find out if the CAN boards connected to the EMS are visible, following the discover procedure: https://icub-tech-iit.github.io/documentation/icub_firmware/firmwareupdater/firmwareupdater/#discover_1

I would ask you to do this procedure on the robot head, for the ems4 boards with IP: 10.0.1.1, 10.0.1.2, 10.0.1.3 and 10.0.1.4.

When you are able to see all CAN boards, I would ask you for a screenshot of the situation!
It would also be useful to have a video of the robot start-up during the calibration phase of the yarprobotinterface.

Please, send us feedback and if you have any questions feel free to ask!

olivierrochel-inria commented 6 months ago

Hi @AntonioConsilvio , thanks for your answer.

Here is the screenshot from the "discover procedure", let me know if this is what you need.

I'll send the video asap.

olivierrochel-inria commented 6 months ago

Hi again, here is the video.

The first few seconds are missing (but I think the calibration phase is there). The right arm look a bit loose to me (The hand even touched the leg during calibration, ooops), it might be related to a tendon that slipped and that we had to put back in place during remounting. The left arm start to move at t=90s approx.

https://github.com/robotology/icub-tech-support/assets/18719389/d80115e7-1ac7-4ddc-8bb6-79bce8f7d2a7

Bonus: screenshots of motorgui showing joint states after calibration for left and right arms

motorgui_right motorgui_left

AntonioConsilvio commented 5 months ago

Hi @olivierrochel-inria, I have a few tips to give that might help decrease problems during calibration (I don't know if they will solve them all).

First, make sure that the configuration files inside the robots-configuration folder are aligned with the new changes made after the maintenance intervention, see PR:
- https://github.com/robotology/robots-configuration/pull/618

Secondly, the starting position of the robot may negatively affect the calibration. In fact, the position of the shoulders (especially the YAW) before the yarprobotinterface is launched should be approximately as follows:

So, the YAW of the shoulders around 0°, if possible the Roll slightly away from the torso and the Pitch slightly forward (around -10° should be fine).

Please, send us feedback and if you have any questions feel free to ask!

olivierrochel-inria commented 4 months ago

Hi, Back in the lab after a while, quick update:

starting with the pose you suggested results in less errors, and very symmetrical errors on both left and right arms, which suggests (to me) a software issue.
The calibration values are ok, but were patched by hand - our system seems to be slightly (!) outdated and there's an error about incompatible firmware in the logs (I missed it somehow... and it was not in the log file I uploaded above, as yarplogger was connected 1s too late for it to appear).

So unless you tell my otherwise, my plan is to start again with a fresh, up-to-date install (will keep the old one as a fallback), try again, cross fingers, and keep you posted. Thanks, O.

AntonioConsilvio commented 4 months ago

Hi @olivierrochel-inria! The update should be possible. Sometimes it can be a bit of a complicated process, but it shouldn't be a bad choice. However, the robot should also work without an update, which is why I ask you again:

A new logger, which with the current starting position of the robot (mentioned in this comment) should have fewer errors and thus be more readable.
A video of the robot start-up during the calibration phase of the yarprobotinterface.
A screenshot of the yarpmotorgui of the joints not working properly.

Please, send us feedback and if you have any questions feel free to ask!

olivierrochel-inria commented 3 months ago

Hi again, I just tried to prepare the new video and the logs you asked for (no update yet, exact same setup as last time) - unfortunately the calibration did not start this time (log attached). Not sure about the reason... I tried 3 times without success. log_pc104_yarprobotinterface_1433.txt

AntonioConsilvio commented 3 months ago

Hi @olivierrochel-inria, analysing the logger, there could be two problems.

It seems that the robot is unable to discover the CAN boards under CFW2.

For this I ask you to run the FirmwareUpdater app on the robot head (entering from terminal with -X being a graphical application). Then select the option there should be called CFW2<0>, and then press discover.

Please send us a screenshot of what you see.
There seems to be something wrong with the right hand MAIS (the board that controls the fingers). However, it seems to be responding. So you have to be able to start the yarprobotinterface to understand the problem.

So, for now, I would focus on problem number 1, because I think that's what is blocking the robot from starting.

Please, send us feedback and if you have any questions feel free to ask!

olivierrochel-inria commented 1 month ago

Hi,

Here is the screenshot, seems to confirm your diagnostics.

cfw2

robotology / icub-tech-support