ergoCub 1.1 S/N:001 – neck cable broken

GiulioRomualdi commented 3 months ago

Robot Name 🤖

ergoCub 1.1 S/N:001

Request/Failure description

After https://github.com/robotology/icub-tech-support/issues/1768 @AntonioConsilvio fixed the neck we performed some walking experiment and the neck got broken again

Detailed context

See the picture of the neck cable

Additional context

I add the video of the robot while walking

https://github.com/robotology/icub-tech-support/assets/16744101/6e11edb0-2556-46ec-a58e-7d37cad17c9c

How does it affect you?

No response

github-actions[bot] commented 3 months ago

⚠️ Detected missing mandatory information

Hi @GiulioRomualdi 👋🏻

Some of the following points need your attention.

You are required to:

[x] enter the Robot Name.
[x] give the issue a meaningful Title.
[ ] provide a Detailed Context (at least 100 characters).

_{Please, mark the points above as solved once done.} [^1]: To get to know the allowed options for the dropdown fields, you may peruse the template.

pattacini commented 3 months ago

Hi @GiulioRomualdi

Better off reducing the safety threshold you've been employing for the position-direct control of the neck, passing from 8 deg to let's say 4 deg.

We could also look at the reference trajectories sent to the neck to verify if this threshold has ever played its role.

GiulioRomualdi commented 3 months ago

Hi @pattacini, to better understand what happened on the robot, I decided to plot the joint tracking, current, and PWM on the neck. Here are some plots:

neck pitch

The following plot shows the neck pitch. The blue line represents the desired quantity we set as a position directive, while the orange one depicts the measured joint position. At t = 38s, the robot starts walking. Despite the required joint position being almost constant, the error increases.

The neck broke at t = 47.29s without any abrupt change in the reference.

neck roll

This plot is similar to the neck pitch.

neck yaw

Unlike the neck pitch and roll, the tracking of the yaw is better.

PWM

Currents

When the neck broke, there was a spike in the current. We may consider limiting the maximum amount of current that the joint can deliver. In this case, the tracking will worsen, but at least we can preserve the joint.

cc @DanielePucci @S-Dafarra

pattacini commented 3 months ago

Nice reporting @GiulioRomualdi 👍🏻

The errors on the pitch and the roll are larger because the mechanism is significantly different from the yaw, although they are in the range of 1.7 deg, at first sight, and are probably mainly caused by the vibrations due to the walking. Just speculating.

Your reference inputs seem ok: no big jumps 👍🏻

True that there's a spike in the current of the roll, but there's also a command voltage rising up to > 75% for the same joint and then falling to -100%, whereas the pitch remains kind of more limited.

It'd be interesting to have a look at that to verify if those spikes are a mere consequence of the breakage cc @sgiraz.

We may consider limiting the maximum amount of current that the joint can deliver. In this case, the tracking will worsen, but at least we can preserve the joint.

It could be a viable workaround.

cc @maggia80

pattacini commented 3 months ago

A couple of comments more.

[!important] MC4Plus doesn't provide current control meaning that the safety action that we can undertake here is to limit the PWM output and not to fiddle with the current limits.

cc @GiulioRomualdi

It'd be interesting to have a look at that to verify if those spikes are a mere consequence of the breakage cc @sgiraz.

The spike in the current could stem from the fact that the tendon broke up on the part of the coupled mechanism that is controlled by one specific motor only, say M0 (or M1), but the other motor M1 (or M0) still sees the opposite part of the mechanism engaged and thus is asked to deliver high current because of the large position error.

To verify this hypothesis, @sgiraz, we could check whether the motor that underwent the spike in current is the one corresponding to the part of the mechanism opposite to that of the broken tendon.

Don't know if I've been clear enough 😄

cc @Fabrizio69

sgiraz commented 3 months ago

@fbiggi and @AntonioConsilvio are going to perform tendon replacement tomorrow morning.

[!Important] Before removing the broken tendon they will identify its location.

cc @pattacini

fbiggi commented 3 months ago

The broken tendon is the one driven by the motor placed in the bottom-rear of the neck. It is described as the tendon A in : https://icub-tech-iit.github.io/documentation/icub_tendons/neck/#23-route-cable-a. The exact breaking point of the tendon will be identified tomorrow morning. See an overview of the mounting points of the tendon is the image below.

@Fabrizio69 @sgiraz @pattacini

pattacini commented 3 months ago

Thanks @fbiggi 👍🏻 We will use this info to proceed further with the investigation.

fbiggi commented 3 months ago

Together with @AntonioConsilvio we replaced the tendon The "A" tendon ruptured at a length of 300 mm The last image identifies its location

Cattura 20240313_121350 20240313_121340 20240313_075608(0)

@Fabrizio69 @sgiraz @pattacini @maggia80

pattacini commented 3 months ago

I aimed to recreate this condition with our Digital Twin 🔒 (DT).

🔲 Partial results

https://github.com/robotology/icub-tech-support/assets/3738070/c36484ab-2496-4043-a18c-928e6dd602df

At t = 2 s I simulated a fault by disconnecting the torque delivered to the revolute joint motor_revolute0 in the RC_IIT_10_A_004 subsystem. Essentially, I wasn't able to manipulate the tendons directly, so I tried to stop delivering the torque of M0.

🔳 Naming convention of motors M0 and M1 in the DT:

M0	M1

The profiles of pitch and roll rise up as in the logs posted from the real system. Obviously, the voltage V0 and the current I0 undergo most of the big changes whereas V1 and I1 stay reasonable.

A few points to inspect.

🔲 The Motor

The broken tendon is the one driven by the motor placed in the bottom-rear of the neck.

This should M1 instead, according to the DT naming convention, correct? @sgiraz @fbiggi

🔲 The Tendons

In the real system, we have 2 tendons: A and B.

This is the tendon circuit in the DT:

I've never really gone into these fine details, I might be wrong, but it looks we have one single tendon circuit.

Perhaps, @mfussi66 and/or @salvi-mattia can add some comments here.

🔲 The electrical quantities (V and I)

For the DT, we have the voltage and the current of the motors.
In the logs from the real system, voltages and currents are maybe remapped to the pitch and roll joints, somehow?

We'd need to check if these quantities are actually remapped or, instead, copied out to the joints' in their order while retaining their meaning for the motors' order.

cc @Nicogene

mfussi66 commented 3 months ago

Perhaps, @mfussi66 and/or @salvi-mattia can add some comments here.

The one-loop solution was the most simple to implement with the best results at the time. Alternatively, one could actually mimick the real mechanism by using two cables, attached to the ends by anchors. At the time, I had problems understanding where to attach those ends. Ideally, they should be connected to the "grains" components, but I had no idea how to make the cable do those weird twists and turns.

pattacini commented 3 months ago

The one-loop solution was the most simple to implement with the best results at the time.

Clear 👍🏻

sgiraz commented 3 months ago

Hi @pattacini,

After consulting @ale-git, we can confirm that from the firmware side, the Voltage and Current parameters are forwarded to the yarprobotinterface via UDP without doing any king of coupling. Just for reference (see: https://github.com/robotology/icub-firmware/blob/5c1d78a70728554eec8d2df3f7ada126bbb2b190/emBODY/eBcode/arch-arm/embobj/plus/mc/Motor.c#L1100-L1117)

If these electrical values are not manipulated (coupled) by yarp and/or icub-main the resulting pitch, roll and yaw have to be considered (logically mapped) as M0, M1 and M2.

[!Note] As discussed F2F we have to consider that, in a coupled system, when a motor lose the connection w/ the transmission the other motor needs to compensate the load increasing the current absorbed.

pattacini commented 3 months ago

Ok, we cannot use the DT in this particular case because the tendon circuit doesn't copy 1:1 the physical mechanism.

From the observations and the analysis we can argue that:

The tendon A is the one that broke.
The motor M1 is the one driving the tendon A.
The high voltage and high current values reported in the plots do refer to motor M1 (roll is the second joint, hence the motor is the second in the order).
After the fault, motor M1 should see a fairly low load compared to motor M0. Thus, we may expect a high current on M0, instead.
At any rate and regardless of the currents, both M0 and M1 should receive high input PWM because of the large position errors. However, only M1 receives high PWM commands. M0 receives similar commands after a while.
From the plots, it's not clear what happens to the position references after the fault though. Perhaps, @GiulioRomualdi can shed light on this. This would allow us to verify the amount of position errors.
Also, @GiulioRomualdi, do you have the logs of the motor positions (not just the joint positions)?

cc @maggia80 @sgiraz @mfussi66 @Nicogene @fbiggi

pattacini commented 3 months ago

Hi @GiulioRomualdi

Also, @GiulioRomualdi, do you have the logs of the motor positions (not just the joint positions)?

By any chance, do you have the whole log to share here? Something like the archives you store on Hugging Face?

pattacini commented 3 months ago

By any chance, do you have the whole log to share here? Something like the archives you store on Hugging Face?

@GiulioRomualdi handed me the data.

pattacini commented 3 months ago

From the https://github.com/robotology/icub-tech-support/issues/1773#issuecomment-1988494371, it's not clear what happens to the position references after the fault though.

The log doesn't contain data on the commanded position setpoints.

do you have the logs of the motor positions (not just the joint positions)?

Pitch and roll motor positions are not retrieved correctly, apparently. We do have just the motor position of the yaw joint.

pattacini commented 3 months ago

In #1781, the neck cable broke up mostly in the same way as reported here: after a short while, when lifting the left foot.

AntonioConsilvio commented 3 months ago

In #1781, the neck cable broke up mostly in the same way as reported here: after a short while, when lifting the left foot.

In addition to always breaking during the same movement (when the lifting of the left foot), we are still verifying the tendon breaking point, which is always the same, about 2.8cm (these are two different tendons):

cavo A

CAVO A 1

For now, we have only been able to verify the breaking point of tendon "A''.

cc @Fabrizio69 @pattacini @Nicogene @GiulioRomualdi

pattacini commented 3 months ago

Seems a quite important piece of information that we can use to double-check the components at stake in the CAD.

pattacini commented 3 months ago

@fbiggi reported that this is the zone where tendon A breaks:

The tendon breaks at 28 mm.

cc @LawProto

salvi-mattia commented 3 months ago

if we suppose the tensioner to be in the middle of its range. 28mm along the cable should be as shown in picture wich appears not to be any parcticular place. do we know where was the tensioner approximately placed?

immagine

pattacini commented 3 months ago

The geometry with which the cable at the length 2.8 cm interacts may vary during operations because of the movements of the pulleys.

Therefore, @salvi-mattia and I were thinking of making a sign on the cable already mounted and then visually inspecting the parts the tendon will "interact" with while moving the neck.

Even though the movement is quite limited, we could obtain some unexpected insights.

S-Dafarra commented 3 months ago

Other related ticket: https://github.com/robotology/icub-tech-support/issues/1783 (not when walking, but it was shaking)

pattacini commented 3 months ago

We decided to proceed as follows:

Replace the two tendons with those with a slightly larger diameter of 0.9 mm.
Inspect the physical mechanism to verify that the distances stated in the CAD among the various pulleys are respected.
Make a sign on the tendon A at the correct distance of 28 mm.
Carry out tests. In details:
- Let the neck move via robotmotorgui and verify the interaction of the marked point with the mechanics.
- Apply vibrations to the neck (manually or semi-automatically) and record a movie with the high frame rate camera to spot possible critical issues.

AntonioConsilvio commented 3 months ago

Hi! @fbiggi and I replaced the tendons with larger ones (0.9mm), see the comment.

We made a black sign on a tendon A at 28mm, and here we can see the interaction of the marked point with the mechanism:

Pitch:

https://github.com/robotology/icub-tech-support/assets/114915464/a7b62a06-bfcb-414f-bfb2-5848cae8e2c4

Roll:

https://github.com/robotology/icub-tech-support/assets/114915464/539e6160-c57b-4be6-81a8-962428c0a047

High Framerate slowmotion (with manual oscillation):

https://github.com/robotology/icub-tech-support/assets/114915464/de6fd8a0-e2d7-4d02-8ef2-f3b01b846970

cc @maggia80 @Fabrizio69 @pattacini @Nicogene

pattacini commented 3 months ago

The new tendon prototypes seem to be working more robustly, being also less flexible.

While the pitch bends forward, the length of the tendon marked with the sign (where it broke 3 times in a row) gets very close to the point where the tendon itself leaves the first pulley to engage the second pulley.

The frontal pulley was also kind of worn out on the border. We repolished it with sandpaper.

pattacini commented 3 months ago

Hi @AntonioConsilvio @S-Dafarra

Probably, we can close this long-standing issue. If we'll have further problems, we can refer to a brand-new ticket, possibly citing this.

AntonioConsilvio commented 3 months ago

Great @pattacini!

Thank you all for the support and feedback, hopefully the new tendons will bring an improvement!

robotology / icub-tech-support