Chapter 3: Markless AR, unexpected behaviour

[oliw]

Hi, Have built the application on latest Ubuntu with OpenCV 2.4.5 and am getting unexpected results.

See youtube screencast here: http://www.youtube.com/watch?v=aTwJZmLft8w

Any ideas what is causing this? Could it be to do with my webcam?

It always eventually just crashes with 'The program has unexpectedly finished.'

[BloodAxe]

Hi,

A crash can be a result of empty homography matrix returned by OpenCV. In the latest (2.4.5) release there was a "breaking change" - cv::findHomography can return an empty matrix. But this is not documented so far. So you may want to check the homography before doing the rest of processing.

[oliw]

Hi, indeed that seems to be the cause of the crash. Is there a quick fix you suggest for this?

it doesn't explain the erratic showing of cubes. Do you have any idea what could cause this? NB. It isn't my webcam because I get the same behaviour when using a video recorded from an alternative camera.

[arthur1026]

I think the erratic showing of cubes is due to the false positive detection of patterns, which means the program thinks part of your scene is the pattern but in fact it is not.

[oliw]

Yes I believe so, I am experimenting extending the program to use the ORB descriptor matcher (rotationally invariant) and also to be capable of using multiple images as a training set. I guess I shall also have to devise a stronger means of false positive detection.

[oliw]

If the author agrees that the presence of this amount of false positives is to be expected then please feel free to close the issue.

[BloodAxe]

Unfortunately we cannot eliminate false-positive matches at all. Descriptor matching stage gives the closest matches between two descriptor sets. It's not guaranteed that matches with minimal distance will be the right ones. The number of false-positive matches depends on the algorithm of feature detection and descriptor extraction. In general, best results can be achieved by using scale and rotation invariant descriptors like SURF. But anyway, after feature matching an outlier removal step is necessary. It can be done via "cross-match test" or "ratio test". A RANSAC-based homography estimation is also very good for finding a good set of matches. I recommend to add small check of the computed homography. From my experience this eliminate many "wierd poses":

bool niceHomography(const cv::Mat_<double>& H)
{
    const double det = H(0, 0) * H(1, 1) - H(1, 0) * H(0, 1);
    if (det < 0)
        return false;

    const double N1 = sqrt(H(0, 0) * H(0, 0) + H(1, 0) * H(1, 0));
    if (N1 > 4 || N1 < 0.1)
        return false;

    const double N2 = sqrt(H(0, 1) * H(0, 1) + H(1, 1) * H(1, 1));
    if (N2 > 4 || N2 < 0.1)
        return false;

    const double N3 = sqrt(H(2, 0) * H(2, 0) + H(2, 1) * H(2, 1));
    if (N3 > 0.002)
        return false;

    return true;
} 

[arthur1026]

Hi BloodAxe,

Is there any geometric meaning behind the mechanics of niceHomography() function ? How did you choose the threshold values of det, N1, N2, and N3 to check whether the pose is weird or not? I agree it would definitely help, and I just want to know the reason behind it.

Thanks a lot!

[BloodAxe]

Unfortunately we cannot eliminate false-positive matches at all. Descriptor matching stage gives the closest matches between two descriptor sets. It's not guaranteed that matches with minimal distance will be the right ones. The number of false-positive matches depends on the algorithm of feature detection and descriptor extraction. In general, best results can be achieved by using scale and rotation invariant descriptors like SURF. But anyway, after feature matching an outlier removal step is necessary. It can be done via "cross-match test" or "ratio test". A RANSAC-based homography estimation is also very good for finding a good set of matches. I recommend to add small check of the computed homography. From my experience this eliminate many "wierd poses":

bool niceHomography(const cv::Mat_<double>& H)
{
    const double det = H(0, 0) * H(1, 1) - H(1, 0) * H(0, 1);
    if (det < 0)
        return false;

    const double N1 = sqrt(H(0, 0) * H(0, 0) + H(1, 0) * H(1, 0));
    if (N1 > 4 || N1 < 0.1)
        return false;

    const double N2 = sqrt(H(0, 1) * H(0, 1) + H(1, 1) * H(1, 1));
    if (N2 > 4 || N2 < 0.1)
        return false;

    const double N3 = sqrt(H(2, 0) * H(2, 0) + H(2, 1) * H(2, 1));
    if (N3 > 0.002)
        return false;

    return true;
} 

There is no magic, just a littel trigonometry behind.

const double det = H(0, 0) * H(1, 1) - H(1, 0) * H(0, 1);
if (det < 0)
    return false;

So in the first check we compute the determinant of the 2x2 submatrix of homography matrix. This [2x2] matrix called R contains rotation component of the estimated transformation. Correct rotation matrix has it's determinant value equals to 1. In our case R matrix may contain scale component, so it's determinant can have other values, but in general for correct rotation and scale values it's always greater than zero.

const double N1 = sqrt(H(0, 0) * H(0, 0) + H(1, 0) * H(1, 0));
if (N1 > 4 || N1 < 0.1)
    return false;

To understand other checks i write the simplified form of homography matrix:

( s * cos(t), -sin(t),    tx)
(     sin(t), s * cos(t), ty)
( px,         py,         1)

It's an approximation of the homography matrix, but in general the top-left [2x2] matrix represents scale and rotation, tx and ty - translation and px, py - projective transformation.

These thresholds were choosen empirically.

[oliw]

I am working on a modified version of this code which uses ORB to detect and extract feature descriptors. I am using the Brute Force Matcher to do matches. I am using Book covers as patterns and found that typically the presence of each pattern caused at least 100 matches. In PatternDetector::refineMatchesWithHomography() the default minimum number of matches allowed in order to report the presence of a pattern is 8. If I increase this value to say 25, I find I get a lot less false positives!

[arthur1026]

I found a trick to make the detection pattern more robust: resize the image pattern to a resolution around 640x480. It works for me well.

[kghalieh]

thanks or your code! I cant understand the N's I think that N1 and N2 limites the scaling between 0.1 and 4 but what about px,py what is the theroy behind it