head spirals out of control when using iKinGazeController

[AlexAntn]

When using the iKinGazeController (like, for example, the red ball demo), the icub head starts spinning out of control, forcing us to cut the power to the robot.

Issuing normal commands to the motors still works fine, and the head is still callibrated. The issue happens only when tracking an object.

A temporary solution was to reduce the rate that commands were sent, but it makes for very poor tracking. This problem resurfaced when we tried to run the red ball demo (without changing anything in the code - so no change in the command transmission rate)

Any idea what might have happened?

[pattacini]

Hi @AlexAntn

My hunch is that the problem is lurking at a lower level with respect to that of iKinGazeCtrl module. Some deeper inspections are likely required.

@giuliavezzani could you get in touch with @AlexAntn to try nailing down the issue?

[giuliavezzani]

Hi @AlexAntn,

let's see if there is a low level problem.

• What are the PID values of the head joints?
• What is the firmware version of the head?

[AlexAntn]

Hello @giuliavezzani ,

Our icub is version 1.2.0, the firmware build is 87 (not sure this answers about the firmware? how can I check it otherwise?)

The PID values are in the following file: headPIDValues.txt

Thanks for the help!

[giuliavezzani]

Hi @AlexAntn,

sorry for the late reply. The PID values seem fine. Also, if you manage to launch the yarprobotinterface without errors, the firmware version should be correct.

Then, I suggest you to check the cables of the neck. Sometimes, this kind of problem happens when the neck cable is a bit loose. Wrong values might be read from the encode, the neck spins faster to keep up with the requests and it does not reach the desired position.

Let me know if it helps!

[AlexAntn]

hello @giuliavezzani !

I checked the black cables around the neck (the ones connecting to the encoders), they all seem to be properly connected (gentle pulls don't dislodge them anyway...)

I haven't seen anything out of place regarding the neck.

So, we opened the head of the icub, and also found nothing out of the ordinary.

I am attaching some pictures of the head boards:

A cable we fixed a (long) while ago:

A loose cable (that, as far as I know, has always been loose):

three perspectives on the control boards:

Maybe a skype call is in order to sort this out, if you have the availability?

Thanks in advance!

[giuliavezzani]

Let's ask if @julijenv sees something strange! Otherwise, if the neck is fine, we should think about how to solve the problem from a higher level.

[giuliavezzani]

Hi @AlexAntn , it has been just told me that @julijenv is not at work this week. He should be back on next Monday. Let's wait for his reply, otherwise we will design an experiment to see where is the problem.

If you are in a hurry and you cannot wait for next Monday, we will do it before!

[julijenv]

Hi @AlexAntn , when @giuliavezzani was asking to get some pictures of the neck I think she was mentionning the mechanincal part of it, not the electronic part. could you check the base of the neck and send us some pictures. I am available on skype, please get in touch. skype ID is icub.support

[dhgarcia]

Hi @julijenv, @giuliavezzani Alex is away, I'm uploading some pictures of the neck, is this what you wanted to see?

[julijenv]

Hi @AlexAntn and @dhgarcia , can u upload a video of whats going on and a logfile? it is quite hard to understand without any logfile, the picture doesnt show much, everything looks alright. Did you try to move the head back and forth by hand (when motors switch OFF) and see if the 2 pulleys (the right and the left - picture 2 and 4 from last mail). Let me know,

[AlexAntn]

hello @julijenv and @giuliavezzani !

We have finally ran some tests with the iCub.

We have checked 4 ports:

• /imuFilter/inertial:o
• /icub/head/state:o
• /icub/head/stateEtx:o
• /icub/inertial

We have saved the data read from these ports into files which are uploaded here.

When the icub is stopped, and not doing any tracking, we get the following: outputHeadState.txt outputHeadStateExt.txt outputIcubInertial.txt outputIMUFilter.txt

We then tried running the icub red ball demo, with 3 different periods for the module. The demo available online has a thread period of 20 ms, which leads to very dangerous behaviour. At 40 ms, it also has some dangerous behaviour, but not as bad, while at 75 ms it behaves reasonably well (although very slow).

the output files for those 4 ports: outputHeadStateExtTracking.txt outputHeadStateTracking.txt outputIcubInertialTracking.txt outputIMUFilterTracking.txt

And a video of the icub behaviour at 20 ms: VID_20170714_163125.mp4.zip (enclosed on a zip file due to file type).

A zip file containing all experiments we did (75ms, 40ms, 20ms and Idle/stopped) icubHeadIssue.zip

Are these the log files/ports you wanted us to monitor? If not, which ports should we be monitoring and creating the log files for?

Thanks for the help!!

[pattacini]

Hi @AlexAntn

A bunch of considerations:

1. How are you launching iKinGazeCtrl?
   • With which parameters?
   • On which machine?
2. Open a console on the above machine and:
   • If no parameters are given, note the last output line of: yarp resource --context iKinGazeCtrl --from config.ini
   • If a file is specified (e.g. iKinGazeCtrl.ini), note the last output line of: yarp resource --context iKinGazeCtrl --from iKinGazeCtrl.ini
   • What is that output?
3. Post here the content of the file found by the command above.

Finally, perform the red-ball demo by launching: iKinGazeCtrl [same options above] --imu::mode off.

This way we disable the IMU that might be not functioning properly. Inspect the messages printed by iKinGazeCtrl to see if IMU gets actually disabled.

[AlexAntn]

hello @pattacini

ok, so point by point:

1. iKinGazeCtrl:

   • We are launching the iKinGazeCtrl with "--headV2 --from config_imu_filter.ini"
   • We launch the iKin modules on a dedicated desktop computer

2. console commands:

   • output: ||| finding file [from] "/home/icub/icub-main/build/share/iCub/contexts/iKinGazeCtrl/config_imu_filter.ini"

3. content of file: [cameras] context cameraCalibration file icubEyes.ini [imu] source_port_name /imuFilter/inertial:o

We tried running the red ball demo without the IMU, and it worked properly! The problem seems to be from the IMU.

We checked the output on iKinGazeCtrl, and it does indeed seem to have disabled the IMU properly: [[01;33mWARNING[00m]IMU data will be not received/used

The output file: outputiKinGazeCtrl.txt

Thanks for the help!

[pattacini]

It's great that we narrowed down the problem @AlexAntn Thanks for testing.

cc @julijenv @giuliavezzani

[giuliavezzani]

Great! Thanks @pattacini for the help!

[AlexAntn]

hello @pattacini , @giuliavezzani , @julijenv !

Yes, it is good that we found the problem, now we just need to find a way to fix it! :D

On this note, how should we proceed? I suppose this all meant the IMU is damaged/not functioning properly somehow, so we should replace it?

Thanks for the help!!

[pattacini]

Don't know if we have somewhere stored specific tests to evaluate the status of an IMU device. @julijenv do we?

[julijenv]

Hi @AlexAntn ,

from this line below, it would be nice to see the firmware or software developper to know what is really the meaning:

[[01;33mWARNING[00m]IMU data will be not received/used

I dont know if it means that the IMU is not functionning or if you didnt open correctly the port to talk to the IMU. I personnaly dont know if it exists the possibility to test the data coming out of the IMU - I guess that a port streams the output (filter-imu something like that), maybe plotting it would let us know if the values are correct. Is it possible that the connector has been slightly disconnected from the pc104 or worst - unplugged or misplaced? her eis a picture of its position on the pc104 in the top left corner of the seeing from the back of the robt head. If correctly plugged and in its position, let's hypotise that you have a non working IMU, it would be nice to proceed by checking the wires from the IMU connector up to the pc104 - by making a continuity test on this cable. If the cable connecting the IMU to the pc104 doesnt work, in this case we would go to the last resort, the IMU is broken and you will have to ask for a quotation to me, and later proceed with an order. So hopefully this reply will make us find that your IMU is alright.

[pattacini]

@julijenv no worries about that message; it's expected since I asked to run a red-ball demo while disabling the reading from the IMU at the level of gaze controller.

By the way, it might be helpful to run the red-ball demo again (with IMU control still disabled) and in the meanwhile log the output of the IMU from the port /icub/inertial

[AlexAntn]

hello @pattacini and @julijenv ,

When we took the video of the red ball demo, we kept a record of the ports /icub/inertial and /imuFilter/inertial:o

the outputs are in the following files (this was for a 20ms period - red ball demo default frequency): outputIcubInertialTracking.txt outputIMUFilterTracking.txt

We also recorded these same ports for different speeds of tracking: 40ms Period: outputIcubInertialTracking.txt outputIMUFilterTracking.txt

75ms period (this one works without going crazy while using IMU): outputIcubInertialTracking.txt outputIMUFilterTracking.txt

When idle (not moving, before we start the red ball demo): outputIcubInertial.txt outputIMUFilter.txt

We checked the IMU cable and it is in good state (we did not perform the continuity test yet).

We will run some more red-ball demos later today and try to get some plots of the ports.

Thanks for the help!

[pattacini]

Hi @AlexAntn

Logging the IMU data when the IMU itself is not used turns to be a different story, so yeah please fresh logs will be helpful - especially because the head won't shake.

[AlexAntn]

hello @pattacini , @julijenv ,

We ran some more tests of the red ball demo, without the IMU. here are the fresh logs of /icub/inertial and /imuFilter/inertial:o from this experiment: outputImuFilter.txt outputicubInertial.txt

[AlexAntn]

hello @julijenv , @pattacini , @giuliavezzani

Any updates on this issue?

Thanks for the help!

[pattacini]

Hi @AlexAntn,

I have zero time to assist you these days, 'cause I'm 100% busy with an important demo. Therefore, I didn't go into your logs yet, and I'm afraid it'll take some time to get back to you. Sorry for not being timely.

My only suggestion, for now, is to check if the imuFilter is not going crazy with your device. Did you try to bypass the filter feeding the controller directly with the IMU output?

Within the configuration files you ought to play with the following params:

• imu::mode should be on
• imu::source_port_name should be /icub/inertial (or you could even get rid of this param)
• try also to reduce the value of imu::stabilization_gain

More info on this are in: http://wiki.icub.org/iCub/main/dox/html/group__iKinGazeCtrl.html

[AlexAntn]

hello @pattacini , @julijenv , @giuliavezzani ,

The good news is it is working!

now, the details:

We tried running the iKinGazeCtrl again, using @pattacini suggestions. It was oscillating a lot, but during the test I noticed the right-eye camera was flickering, sometimes presenting black frames. At some point it only provided black frames, so we stopped the robot (it obviously went crazy when it got no decent image).

We traced the problem to a damaged flat cable on the right-eye, and after replacing it the camera worked fine. We did some more tests, and found no issues with the tracking anymore.

We had not detected any problems with the camera before, but it might be that the problem has been there since the beginning, ruining the tracking but still rare enough to not show up on the yarpview.

We have ran the test with the red ball demo several times already, and didn't have to press the red button a single time yet.

Is it plausible that this camera problem would originate the problems with the iKinGazeCtrl? If yes, then I think we can close this issue, since the problem is solved...

[pattacini]

Hi @AlexAntn

I wish I could say your intuition is right but I'm afraid there might be something more.

Usually, the red ball demo employs the left eye for the tracking, unless you specifically dictated to use the right one. In fact, the tracking is purely monocular.

Therefore, there must be something subtle undergoing here, and quite difficult to nail down. Also, I feel we cannot rule out the possibility that an HW problem on a cable did affect the SW driver handling the camera images. In this respect, we have recently introduced a unique server grabber for that purpose.

To sum up, your tracking is running fine now with the IMU enabled; is that right? If so, we can deduce that the IMU was likely not the problem.

[AlexAntn]

Hello @pattacini ,

Yes, the tracking is currently working with the IMU enabled. We tried a couple of times, in different days, and detected no issue (while before it would go crazy almost immediatelly).

We can run some more tests, maybe specifying the right eye camera for the tracking, to see if anything changes?

[julijenv]

CIao @AlexAntn , could you open the head cover (front and rear) and check the if the 3 last joint of the head are well tense?!?!? After calibration look if the movement of those motors can be moved by hand, maybe you have too much slack on those joints due to some screws loose, please check that. let us know Im behind my desktop if ever needed. skype ID is icub.support

[AlexAntn]

hello @julijenv ,

I have checked the motors of the head joints, they all seem to be tense. there is a slight slack on the head roll, but not significant (the cable loses a bit of tension when I push the head left).

I can check in on skype around 14 (15 there) if you are around!

thanks!

[pattacini]

@AlexAntn

I'm afraid the test wouldn't be conclusive...

Perhaps, testing the functionalities of iKinGazeCtrl alone, i.e. without vision involved, would instead help narrow down the problem.

You could do that:

1. Manually, by sending proper commands via rpc.

   yarp rpc /iKinGazeCtrl/rpc look 3D (-0.5 0.0 0.36) look 3D (-0.5 0.2 0.36) look 3D (-0.5 -0.2 0.36) look 3D (-0.5 0.0 0.1) ...

2. Programmatically, by using some sort of naive client. You might consider running the https://github.com/vvv-school/tutorial_gaze-interface tutorial while adapting the control part to run on the real robot and keeping the simulated vision as fake input.

[pattacini]

@AlexAntn

A simpler tutorial feeding the iKinGazeCtrl is: https://github.com/robotology/icub-tutorials/blob/master/src/motorControlAdvanced/tutorial_gaze_interface.cpp.

[julijenv]

@AlexAntn , just ring me on skype, and C if Im around, or else just let me a message on skype Ill answer to propose a meeintg time

[AlexAntn]

hello @pattacini , @giuliavezzani ,

From @julijenv :

From a debugging session with @AlexAntn on skype, we were able to detect some looseness inside the neck and, after resolving this mechanical problem, we succeeded to get a better behavior of the robot when the IMU is not used. When using the IMU, it worked (on the two experiments we ran), although the head oscillated a bit more than without IMU.

The IMU apparently responds and gives values BUT who knows if they are relevant. Someone knows anything about IMU fileter data???

Logs of the test we ran: imuFilterOutput.txt icubInertialOutput.txt outputiKin (1).txt

Edit: We tried running the "look 3D (-0.5 0.0 0.36)" command but it returned "command not recognised". Other commands worked fine (i.e: "get pose head"), but the look command always returned "not recognised".

Thanks for the help!

[pattacini]

Hi @AlexAntn

This simply means that you're not running the latest software.

The look commands have been introduced since https://github.com/robotology/icub-main/commit/684aede2ffc0534395c144d851d464bb54cdd1b3#diff-ce4b92b1a9b6650b0ada895643e256f4, which dates back to last December. My suggestion is, therefore, to update the system and try out calling that command again.

Further, it would be of help to log data using the yarpdatadumper tool, so that it'll be much easier to compare results.

Finally, it would be of extreme help 😛 to rule out the vision part. To this end, you could run two separate sessions - one with IMU, one without - using the tutorial I mentioned earlier.

[pattacini]

Further observations I have:

• I would log the output of:
  • /icub/inertial
  • /imuFilter/inertial:o
  • /icub/head/state:o
• The gaze controller gets configured to the use of head hw version 1.0, which is quite old (I've caught this from the output). Is this version correct? Which head are using?

[pattacini]

From the repo, iCubPlymouth01 should be using the head version 2.0.

I argue that from the location on the system where you launch iKinGazeCtrl it's likely that robots-configuration is not properly installed.

Also, it might be that iKinGazeCtrl is launched using the default configuration file config.ini rather than iKinGazeCtrl.ini.

Therefore, one possibility would be adding the line head_version 2.0 at the top of config.ini.

[pattacini]

Maybe it can be considered fixed by now?

[dhgarcia]

I think we can closed it for now. And reopen if we find something else. We haven't had time to run the tutorial you mentioned or check the problems with the unrecognised "look 3D (-0.5 0.0 0.36)" command.

Once we check this again I could reopen or create a new issue.

Thanks all for the help so far.

[pattacini]

Thank you @dhgarcia for reporting.