aruco: Charuco corners mislocalized because of singular matrices in _interpolateCornersCharucoLocalHom()

[amy-tabb]

System information (version)

• OpenCV => 4.3
• Operating System => Ubuntu 18.04
• Compiler => c++

Detailed description

aruco contributed module, charuco.cpp

Charuco corner interpolation fails very rarely in the the _interpolateCornersCharucoLocalHom() function in charuco.cpp. (This function is called by int interpolateCornersCharuco(..)). The reason for this failure is that at this line 471:

 transformations[i] = getPerspectiveTransform(markerObjPoints2D, _markerCorners.getMat(i));

The transformations[i] matrix is singular; all zero with the (2,2) position = 1.

Steps to reproduce

Call int interpolateCornersCharuco(..) with input such that transformations[i] will be singular.

The first image shows an example of this behavior -- the aruco markers were correctly identified, but the charuco interpolation failed for the reason I mentioned. The second image shows the same image but with the reprojection error as magenta lines for one calibration.

Issue submission checklist

• [ x] I report the issue, it's not a question
• [x ] I checked the problem with documentation, FAQ, open issues, answers.opencv.org, Stack Overflow, etc and have not found solution
• [ x] I updated to latest OpenCV version and the issue is still there
• [ x] There is reproducer code and related data files: videos, images, onnx, etc

[alalek]

Please provide complete minimal reproducer:

• used detection parameters
• original image (lossless .png format is preferable)

[amy-tabb]

Added the requested information.

• problem reproducible with a range of detection parameters, used the default in the code sample below.
• original image image_00003.png included.
• code sample that reproduces the problem.

  
  // altered calibrate_camera_charuco.cpp

include <opencv2/highgui.hpp>

include <opencv2/calib3d.hpp>

include <opencv2/aruco/charuco.hpp>

include <opencv2/imgproc.hpp>

include

include

include

using namespace std; using namespace cv;

int main(int argc, char *argv[]) {

string inputFile = "../image_00003.png";
string outputFile = "../out-charuco.png";

int squaresX = 13;
int squaresY = 28;
float squareLength = 300;
float markerLength = 150;
int dictionaryId = 11;

Ptr<aruco::DetectorParameters> detectorParams = aruco::DetectorParameters::create();

Ptr<aruco::Dictionary> dictionary =
        aruco::getPredefinedDictionary(aruco::PREDEFINED_DICTIONARY_NAME(dictionaryId));

// create charuco board object
Ptr<aruco::CharucoBoard> charucoboard =
        aruco::CharucoBoard::create(squaresX, squaresY, squareLength, markerLength, dictionary);
Ptr<aruco::Board> board = charucoboard.staticCast<aruco::Board>();

// collect data from each frame
vector< vector< vector< Point2f > > > allCorners;
vector< vector< int > > allIds;
vector< Mat > allImgs;
Size imgSize;

{
    Mat image, imageCopy;
    image = imread(inputFile.c_str());

    vector< int > ids;
    vector< vector< Point2f > > corners, rejected;

    // detect markers
    aruco::detectMarkers(image, dictionary, corners, ids, detectorParams, rejected);

    // refind strategy to detect more markers
    //if(refindStrategy) aruco::refineDetectedMarkers(image, board, corners, ids, rejected);

    // interpolate charuco corners
    Mat currentCharucoCorners, currentCharucoIds;
    if(ids.size() > 0){
        aruco::interpolateCornersCharuco(corners, ids, image, charucoboard, currentCharucoCorners,
                currentCharucoIds);
    }

    // draw results
    image.copyTo(imageCopy);
    if(ids.size() > 0) aruco::drawDetectedMarkers(imageCopy, corners);

    if(currentCharucoCorners.total() > 0){
        aruco::drawDetectedCornersCharuco(imageCopy, currentCharucoCorners, currentCharucoIds);
    }

    imwrite(outputFile.c_str(), imageCopy);
}

return 0;

}

![image_00003](https://user-images.githubusercontent.com/25328047/79636421-827a4880-8145-11ea-93bd-55a05b937a18.png)
![out-charuco](https://user-images.githubusercontent.com/25328047/79636425-873efc80-8145-11ea-9ed5-401add9cdd27.png)