torque measure saturated

[serena-ivaldi]

We are observing strange warning messages about a torque measure saturated in the shoulder (joint 0) of the left arm (10.0.1.2) of iCubNancy01. This is the kind of messages:

WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.7889 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 60948

These messages happened when the robot arm was not moving. It had calibrated correctly and there were no errors before. We checked the shoulder joint on the left arm and there was no apparent problem. Also we can't understand the -2.7 and +2.0...

I attach the yarplogger log to see more of them. What is the problem with this torque saturation? Is it a motor problem? Should we get worried?

Thanks for the help!

@dogoepp @misaki43

ypmrprunlog_28_09_2016_03_46_06.txt

[serena-ivaldi]

Dear all

this warning message is appearing again, could you help us solve it?

We start our robot and just launched wholeBodyDynamics for the torque estimation and we have again the same warning message (which I would classify as error, not as warning, because I cannot use the robot in this mode):

3375 2162,440824 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1182 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 11450 3372 2160,441538 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1558 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 11257 3371 2158,432850 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1405 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 11062 3367 2156,423689 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.2233 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 10866 3364 2154,415459 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1641 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 10672 3363 2152,416542 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.0975 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 10480 3357 2150,408760 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.2014 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 10284 3356 2148,399844 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1580 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 10090 3355 2146,390704 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.2230 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 9896 3349 2144,381751 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.2476 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 9709 3342 2142,384424 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.0370 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 9517 3339 2140,377435 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.0581 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 9337 3335 2138,370355 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1369 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 9149 3334 2136,362753 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1638 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 8958 3330 2134,354912 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.0370 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 8773 3326 2132,348412 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1645 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 8587 3323 2130,331039 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.0598 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 8400 3322 2128,324420 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.0804 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 8216

What we see in the yarpMotorGui is that the torque of the elbow in the left arm is saturated at +2 Nm. Moreover, we looked at the end effector wrenches and we can see that they have very high values not consistent with a zero contact situation:

RIGHT ARM END-EFFECTOR WRENCH -19.32172 -22.525813 6.772374 -1.018496 2.249015 4.446692 LEFT ARM END-EFFECTOR WRENCH 19.237975 23.080204 -39.759141 3.028513 -9.803893 -3.99113

We checked the output of the FT sensors in both arms and it seems we have a serious problem with the FT sensors (or their calibration?): now the robot is standing and not moving, and not having contacts, and here are the readings of the FT sensors in the arms:

LEFT ARM FTS: 30.474152 -625.937256 -575.912476 -6.248291 7.962769 -4.989166

RIGHT ARM FTS: 3.297913 0.61554 423.18512 5.69397 0.678406 0.157898

So it seems that we have a serious wrong measure in the left arm FTS, and an apparent wrong measure in the Fz at least in the right arm.

Should we change the FTS?

[julijenv]

Hi @serena-ivaldi , I dont know if the FTS problem is related at some wires problem but i think before starting to open up the forearms it would be wise to ask for some feedback from the user of FTS. @traversaro , @pattacini . Is it possible that the FTS has its matrix of calibration wiped out or changed? is there a way to check the output of every single F and T from the FTS would give us more inside of the problem? last but not the least, can you see the FTS on canLoader?

[pattacini]

I'm not into this matter, sorry.

[traversaro]

Moreover, we looked at the end effector wrenches and we can see that they have very high values not consistent with a zero contact situation:

RIGHT ARM END-EFFECTOR WRENCH -19.32172 -22.525813 6.772374 -1.018496 2.249015 4.446692 LEFT ARM END-EFFECTOR WRENCH 19.237975 23.080204 -39.759141 3.028513 -9.803893 -3.99113

Those values change when you calibrate wholeBodyDynamics? Have you tried to visualize these forces in iCubGui ?

We checked the output of the FT sensors in both arms and it seems we have a serious problem with the FT sensors (or their calibration?): now the robot is standing and not moving, and not having contacts, and here are the readings of the FT sensors in the arms:

LEFT ARM FTS: 30.474152 -625.937256 -575.912476 -6.248291 7.962769 -4.989166

RIGHT ARM FTS: 3.297913 0.61554 423.18512 5.69397 0.678406 0.157898

This values are before offset removal, so we cannot conclude there is something strange only by looking at this raw data. However, to answer @julijenv question, it happened in the past on iCubDarmstadt01 that the calibration matrix for an FT sensor was corrupted (see https://github.com/robotology/icub-firmware/issues/68). This however can be easily checked by using the interface for the calibration of FT sensor, accessible as described in https://github.com/robotology/QA/issues/88 . We recently released a new tool that is going to substitute the canLoader, but I don't know if it has been "officialy" released or not, perhaps @marcoaccame can provide more details.

[serena-ivaldi]

We will try to diagnose if the problem comes from wholeBodyDynamics by plotting the values before and after we launch it. However, I have no experience into changing the offset values with the slides that are mentioned in the QA#88

[traversaro]

We will try to diagnose if the problem comes from wholeBodyDynamics by plotting the values before and after we launch it.

Please note that wholeBodyDynamics is not modifying the value that you can read from the F/T sensor ports at all.

However, I have no experience into changing the offset values with the slides that are mentioned in the QA#88

No need to have experience in the offset removal: this is just an issue describing the use of the canLoader --calib command, that is giving you an easy visualzation of the calibration matrix for a F/T sensor, and then you can check if there is anything is wrong.

[serena-ivaldi]

So I turned on the robot and dumped some data from the FTS sensors ... I turned on wholeBodyDynamics after a while and the values do not change (as Silvio confirms). However, they are wrong since the beginning. I printed in the xlabel the mean of the values. Considering that the robot is not moving and not having contacts (he is on the pole) it looks weird.

To be precise, I dumped the data from /icub/left_arm/analog:o and analogous ports for RA, LL, RL.

@olivierrochel-inria

[serena-ivaldi]

@traversaro since these are values going out the /analog:o port, should I read the raw values of the FT sensor to compare? If it's a problem of the calibration matrix (at low level I assume), when is it changed/overwritten/computed? And if I do the calibration procedure (with the canLoader) how do I change the values (the issue was talking about sliders)? I mean I cannot change the values by trial and error.... sorry I am a bit worried of making big mistakes.

[randaz81]

Hi Serena, just to clarify:

3375 2162,440824 WARNING embObjMotionControl::updateMeasure() torque measure saturated from -2.1182 to +2.0000 on BOARD EB2 IP 10.0.1.2 joint 0, count: 11450

is the wrist pronosupination and NOT the elbow mentioned in:

What we see in the yarpMotorGui is that the torque of the elbow in the left arm is saturated at +2 Nm.

That said,

• robots-configuration\iCubNancy01\hardware\mechanicals\left_lower_arm-ems2-mec.xml indicates 2Nm as the full scale for pronosupination joint, so this is correct.

• wholeBodyDyncamics is responsible for computing the torque of pronosupination joint and it removes the offset when it is started and when a 'calib all' command is sent on its rpc port.

• these offsets are normal and are not compensated on the FT sensor port (i.e. /icub/left_arm/analog:o). They are compensated on the /XXX/left_arm/Torques:o port.

• so far I think that you do NOT have to touch anything on the FT sensor calib matrix.

[serena-ivaldi]

Hi Randaz

Torques:o gives the joint torques, and those are extremely high. Also, 2Nm in the wrist do not make sense (too high). When I turn on the robot, he thinks he has very high external forces at the hands, while they should be zero. So... I don't know what is wrong but these values do not seem correct. What should I check? Is it possible that wholeBodyDynamics has a v2 upper-body and not a v1? Is there a flag for that? Or a way to check that these are ok?

[gabrielenava]

I suspect this problem is not related to wholebodydynamics.

I'm attaching to this issue a .pdf containing the calibration procedure, I don't know if you already have it. In particular, can you open the canloader (window shown at page 13)? should be something like this (the value in the matrix are not realistic in the picture):

If the matrix on the right is full of f, it is corrupted and you have to perform the calibration again. If the offset on the left is too high (> 900) or too low (< 600) it may be necessary to reduce it to 700 and recalibrate the sensors. Can you put the screenshot of the canloader in this issue, anyway?

TP_FT_SENSOR_CALIBRATION.pdf

[serena-ivaldi]

Dear all

we have again a problem of torque saturation, this time in the legs (and we almost killed the robot while doing torqueBalancing!!). The torques on the right leg are more than 2000Nm. We managed to connect to the FTS via canLoader and retrieve the calibration matrix, which seem indeed wrong. However now it seems that all the four FTS of the legs are satured or miscalibrated. See screenshots. What should we do? Recalibrate? How is the procedure done? In the PDF attached by Gabri the calibration procedure is only for the FTS not on the robot... Should we just click on " reset calibration"?

[serena-ivaldi]

Update: we rebooted the robot many times and at some point we went back to normal torque values. Since in all these reboots we experienced many different issues with the boards (some boards unreachable, then re-appering, then many FT sensors saturated, etc) I guess there is a problem with the boards or the way the boards communicate with the FTS... This problem appeared also when @gabrielenava and @pi-q were visiting...

any ideas on that @marcoaccame ? how can we debug that?

cc @julijenv @traversaro @dogoepp @InriaBrice

[marcoaccame]

hi @serena-ivaldi ...

1. for calibration issues on the strain i think that you should not use the calibration window unless you are guided to operate on that by someone of the icub-support team. i think that @julijenv can help you on that.
2. for other problems such as reboots we experienced many different issues with the boards (some boards unreachable, my suggestion is to open another issue and attach all the relevant information (logs, description actions etc.). after that the support team can analyse data and help you.

[marcoaccame]

one more thing ... why do you say that when you retrieve the calibration matrix they seem indeed wrong.?

did you compare the values on the gui with the original values of SN099, SN205, SN101? maybe @julijenv can help you on doing this comparison.

[serena-ivaldi]

Hi @marcoaccame, for the wrong calibration matrix please look at the screenshots I posted few comments before. You can see that the values in the matrix are high, and in the right column you have values that are saturated (655300 and similar). When we turn on the robot, most of the times the values from /icub/PART/analog:o are not " saturated" but only high (ex: 40 Nm in the foot in the air, sometimes with wrong axis), the problem is the dynamics computation because we have joint torques in the legs that are more than 2000 Nm! So maybe the problem is not in the FT sensor calibration, but it seems to me that this is the most likely, and when @gabrielenava was here in visit he also observed this problem.

Could you please tell me where to look for the FT sensor value comparison?

[serena-ivaldi]

I have more info to give you on the FT sensors of the feet. Right now, the right FTS of the foot is unreachable... it does not stream any value. The left foot FTS is streaming values but there is something clearly wrong. The robot is now with the feet on the ground and it is not moving. The plot below shows the force components: Fx Fy Fz = red green blue Since the robot is not moving, the forces should be almost constant, instead they have some pikes and change values abnormally. Does this mean we are reaching the max of the sensor or we need to recalibrate?

[marcoaccame]

Hi @serena-ivaldi, I advise you to refer to @gabrielenava and co. for expected / strange behaviour of FT. About the strain board being unreachable, please ask @julijenv.

[serena-ivaldi]

Hi @marcoaccame and @gabrielenava , I suspect that the saturating behavior of the FTS in the feet (see picture) is due to the wrong calibration matrix. @gabrielenava can you confirm?

[julijenv]

Hi @serena-ivaldi , could you try to send us the serial number of all your FT sensor please ( I succeed to get 3 of them (099, 205, 101 but what about the others), please tell us where are they from (arm or leg (hip or ankle)). thx

[serena-ivaldi]

@julijenv where can I find the serial number?

[fjandrad]

@serena-ivaldi Just before the calibration window. In the canLoader window once you connect to the board you can see that one of the columns has S/N of strain , that is the serial number of the sensor.

[julijenv]

Need to change them to remove that saturation