SFM exits when the head is moving (ARE)

[maxime-petit]

Hi all,

SFM goes into segfault as soon as the head is moving using ARE (during a look or a point command). It crashes as soon as the head is moving. However, SFM is fine when I move the head with yarpmotorgui (joint 0 and 1). Maybe because the moving speed was slower?

The error is: Error: Input must have at least three input vertices Error: Input must have at least three input vertices Segmentation fault (core dumped)

iKinGazeCtrl is launched with --eye_tilt::min 5.0 --eye_tilt::max 5.0 --imu::mode off, in addition to neck_position_control off

Is it a known problem that is due to some wrong calibration, parameters tuning, ... or it is something else?

./Max

[pattacini]

Of course, ARE and iKinGazeCtrl have no impact on SFM seg-fault. It must be just a matter of speed that affects the acquisition.

However, it is an ELAS related error.

Perhaps @GiuliaP has a chance to investigate it. For the time being, try to reduce the speed of the gaze and/or go back to sgbm mode.

Out of curiosity: what's the purpose of --eye_tilt::min 5.0 --eye_tilt::max 5.0?

[pattacini]

Btw, I've never seen this seg-fault before, so I can't reason about its cause yet.

[GiuliaP]

Hi,

I have used the SFM along with ARE to move the head (e.g. with the command track motion no_sacc), to track the closest object in the scene, but I have not observed this error. As @pattacini was saying, it could be a problem related to the speed of the gaze. I am currently in the US for a Summer School and I will be back in 2 weeks, however, I could see if I manage to investigate the problem while I am here. Btw, could you provide the exact way you use the SFM, the ARE, the iKinGazeCtrl (e.g. with an XML + the commands that issue the error) so that we can reproduce the situation? Thanks!

[maxime-petit]

Hi,

--eye_tilt::min 5.0 --eye_tilt::max 5.0 is to keep the vergence of the eyes static.

What is the sgbm mode?

I wil try to check with track motion no_sacc. To reduce the speed of the gaze how should I do that properly?

To launch, I use the iCubStartup.xml (from the template) with the modifications stated previsouly for iKinGazeCtrl. I use the SFM.xml provided also by the stereo-vision repo. I launch ARE without any parameters by hand (for checking purpose). The command sent was rpc commands yarp rpc /actionsRenderingEngine/cmd:io home all look (-0.5 -0.15 0.05)

Same is happening with point (-0.5 -0.15 0.05). home all is fine, it is during the look/point that the crash is happening

[maxime-petit]

Found the --use_sgbm :). I'll try tomorrow!

[pattacini]

@maxime-petit that configuration line is definitely not doing what you want to achieve.

The correct configuration options needs to be passed to ARE and not to iKinGazeCtrl.

Therefore:

1. remove --eye_tilt::min 5.0 --eye_tilt::max 5.0, which might have caused problems while gazing.
2. launch ARE with option --motor::block_eyes 5.0 or put the same option within the motor group of the corresponding configuration file.

Remember that, if you block the eyes, then no_sacc option won't have any effect.

[maxime-petit]

Ty Ugo, I 'll try tomorrow On 19 Aug 2015 19:09, "Ugo Pattacini" notifications@github.com wrote:

@maxime-petit https://github.com/maxime-petit that configuration line is definitely not doing what you want to achieve.

The correct configuration options needs to be passed to ARE and not to iKinGazeCtrl.

Therefore:

1. remove --eye_tilt::min 5.0 --eye_tilt::max 5.0, which might have caused problems while gazing.
2. launch ARE with option --motor::block_eyes 5.0 or put the same option within the motor group of the corresponding configuration file.

— Reply to this email directly or view it on GitHub https://github.com/robotology/stereo-vision/issues/6#issuecomment-132725704 .

[pattacini]

Note for @GiuliaP

Actually, a saccade might cause the seg-fault, due to its intrinsic super-fast movements. I'd check this as the very first experiment.

Note for users

There has been since long a big caveat for the use of saccades along with vision processing. Thus, it may be always convenient to disable saccades when vision algorithms are not that robust against them.

Saccades can be disabled by means of:

1. iKinGazeCtrl configuration option: saccades off.
2. At run-time, issuing the rpc command set sacc 0 to /iKinGazeCtrl/rpc port.
3. At run-time, using the gaze I/F method setSaccadesMode(false).
4. ARE configuration option: --motor::block_eyes <ver>.
5. ARE run-time option: no_sacc.

[pattacini]

@GiuliaP did you have the chance to run the test? @maxime-petit did you verify that disabling saccades can get around the problem?

[maxime-petit]

Hi,

Running ARE with --motor::block_eyes 5.0 solved part of the crashing (during the pointing) but I still have some problems sometimes when going back to home after a pointing gesture.

I will try to add the no_sacc but it should not change anything as you say in your email 6 days ago?

I will also try the use_sgbm this afternoon hopefully

Thanks

[pattacini]

@maxime-petit just launch iKinGazeCtrl with saccades off and you're safe.

[maxime-petit]

Using SFM with use_sgbm is way better : no crash and the object localization is more "stable".

[lornat75]

I hope we all agree that SFM should not crash when the robot performs a saccade, this is clearly a bug which (unless we understand it well) could happen even in other situations (e.g. when the eyes move fast).

Do you have an idea why it crashes? I can only hypothesize that there is a big mismatch between the eyes (or the encoders) that is not well handled. It should be investigated.

Another thing to keep in mind when using the SFM is that we have found (ref: @apaikan) that sometimes images are not synchronized and the quality of the depth map clearly degrades heavily. This is currently under investigation.

[pattacini]

@maxime-petit :open_mouth: Well, no crash is expected since we don't employ elas, but the quality of the reconstructed depth should be better with elas. How do you judge the two cases? Could you post examples of them?

@lornat75 What we know is:

• With elas we get SFM exiting (i.e. not seg-faulting) at this line (triexit() is only a wrapper over exit()). No idea yet, still to be inquired.

[Tariq-Abuhashim]

@maxime-petit These are my thoughts on this issue Libelas may exit for the following reasons:

1. Unlike SFM, which uses the same calibration parameters obtained at start up to rectify the images then texture matching to estimate the depth, libelas runs the visual pipeline from scratch again. It extracts corners, matches them, then matches the texture with a square region around them. One reason it would exit without a segmentation fault is because there are less then 3 support points (or matches). // if not enough support points for triangulation if (p_support.size()<3) { cout << "ERROR: Need at least 3 support points!" << endl; _mm_free(I1); _mm_free(I2); return; } Also, as @pattacini Ugo mentioned, Delaunay triangulation requires a minimum of three support points, or it will exit: if (m->invertices < 3) { printf("Error: Input must have at least three input vertices.\n"); triexit(1); }
2. Disparity estimation requires building a grid, Delaunay triangulation, and performing interpolation. If the grid settings used are incorrect, the code will crash due to segmentation fault. In order to check this, post the settings you using with libelas.
3. As @lornat75 mentioned, since rectification is done using start-up calibration, faults in frames synchronization will cause two problems: A. The images wont match the initial calibration, hence rectification wont make sense, and libelas wont find features or depth map will be inaccurate. B. This happens mainly when the head changes heading. If the head rotates around the vertical axis, and due to faults in images synchronizations, the closest left to right image time-wise may happen to be also closer distance-wise (ie, baseline is shorter than the CAD 68mm). At very short baselines, depth uncertainty will explode, and the depth map will be inaccurate.

My suggestions for temporary solutions are: 1- check Elas settings 2- make sure the scene is texture rich. 3- move the head really slow to reduce the effect of images synchronisation.

Code-wise, this can be fixed by: 1- If less than three (I usually set it to 50) support points, do not exit, but drop frames, and move to next. 2- If baseline less than 68mm, do not triangulate features or estimate disparity, but drop, and move to next frames.

Unfortunately, if this means that you have to drop all the frames, or you wont have enough frames, then online SFM is required, and we are aiming for that very soon.

[Tariq-Abuhashim]

Here are the parameters I used with Elas in MATLAB. The resulted disparity will contain some gaps, but its dense enough I think: param.candidate_stepsize = 2; % step size of regular grid on which support points are matched param.lr_threshold = 50; % disparity threshold for left/right consistency check param.add_corners = 0; % add support points at image corners with nearest neighbor disparities % used in (computerMatchingDiparity) param.disp_min = 0; % minimum matching search disparity (positive integer) param.disp_max = 255; % maximum matching search disparity (positive integer) param.support_threshold = .90; % max. uniqueness ratio (best vs. second best support match energies) param.support_texture = 10; % min texture for support points % used in (removeInconsistentSupportPoints) param.incon_window_size = 10; % window size of inconsistent support point check param.incon_threshold = 5; % disparity similarity threshold for support point to be considered consistent param.incon_min_support = 10; % minimum number of consistent support points (for a point not to be removed) % used in (createGrid), (findMatch), and (elas.cpp) param.grid_size = 1; % size of neighborhood for additional support point extrapolation % over 100 is dangerous, may cause memory error in % matlab % used in (findMatch) param.match_texture = 10; % min texture for dense matching % used in (computeDisparity) param.beta = .01;%.02; % image likelihood parameter (smaller = smoother) param.gamma = .1;%3; % prior constant (smaller = smoother) param.sigma = 1; %1 prior sigma (smaller = smoother) param.sradius = 2;%2; % prior sigma radius % used in (removeSmallSegments) param.speckle_size = 100; % maximal size of a speckle (small speckles get removed) param.speckle_sim_threshold = .95; % similarity threshold for speckle segmentation % used in (gapInterpolation) param.ipol_gap_width = 10; % interpolate small gaps (left<->right, top<->bottom) % used in (Elas::process) param.subsampling = 0; % process only each 2nd pixel (1=active) param.postprocess_only_left = 0; % saves time by not postprocessing the right image param.filter_adaptive_mean = 0; % optional adaptive mean filter (approximated) param.filter_median = 0; % optional median filter (approximated)

[maxime-petit]

Hi,

I think SFM quit because of the reason 1, or at least we have this error message appearing twice before segfault. Because we experiencing some jerkyness in head movements, leading to oscillations during them (see https://github.com/robotology/icub-support/issues/48), I guess the images can be "blurry" and thus support points are not present anymore? Or lead to 3.B and fault im image frame synchronization? I will try to record the camera flows.

For suggestion 1, we have not changed any parameters in elas it should take the ROBOTICS settings (stereo-vision is up-to-date). Should I try with the parameters you are using? I will test with a richer texture scene, meaning basically more objects right? Currently we were testing with 3 objects. I will also reduce head movements speed and let you know.

Thank!

./Max

[Tariq-Abuhashim]

@maxime-petit Which parameters to use? it depends. The first few parameters are the most related to the number of matches. Not only image blurring may affect that, but also focus and distance to objects

param.candidate_stepsize = 2; % step size of regular grid on which support points are matched param.lr_threshold = 50; % disparity threshold for left/right consistency check param.add_corners = 0; % add support points at image corners with nearest neighbor disparities % used in (computerMatchingDiparity) param.disp_min = 0; % minimum matching search disparity (positive integer) param.disp_max = 255; % maximum matching search disparity (positive integer) param.support_threshold = .90; % max. uniqueness ratio (best vs. second best support match energies) param.support_texture = 10; % min texture for support points

Also, I assume that you received two identical exit messages, one for the left and one for the right images, since both are processed. Texture rich means more corners in the image at good resolution, and less reflections. Its hard for me to tell, because libelas uses Blob detectors, which they considered as a fast version of a corner detector. I use a KAZE detector, which produces more features in normal conditions, I think Vadim may have imported it into iCub.

[pattacini]

It would be preferable to solve this problem without exposing users too much to the intrinsic complexity behind the working principles of libelas. Many parameters and possibilities will make user's life not easy.

Let's be pragmatic and try to reproduce the problem here. We will then investigate it and design counteractions for that.

[Tobias-Fischer]

Hi Ugo @pattacini, that sounds great. @maxime-petit and I will use SGBM for our tests in the meantime, and will wait for an updated version of SFM with improved usage of the libelas library. I guess that dropping frames and returning a disparity image with invalid disparity (or returning the last disparity image again) sounds like an okay-ish solution without too many changes in libelas. Thanks for looking into that issue!

Best, Tobias

[pattacini]

@GiuliaP when do you have time to discuss how we should cope with this issue?

[GiuliaP]

@pattacini I will come back to IIT tomorrow and eventually all next week, if it's ok for you.

[pattacini]

Perfect, thanks. Let's do it next week, 'cause tomorrow'll be heavy day of meetings for me.

[GiuliaP]

Fine, thank you!

[pattacini]

Good news: @GiuliaP has an upcoming PR that should solve this issue.

[Tobias-Fischer]

:+1:

[pattacini]

8 seems to solve this issue.

Thanks @GiuliaP.