How to identify a marker？

[hechaochao]

• I am a newbie in slam, I have some questions to ask. I want to introduce a marker into slam that uses the marker coordinate system as a reference when rendering objects.
• My idea is to refer the marker as the oxy plane, the center point of the marker as the origin of the marker coordinate system by specifying the pixel coordinates of the four corners of marker then normalize it, so we can get the three-dimensional coordinates of point on markers like (x , y, 0). Extracting the features from the marker, looking for the matched MapPoint to find a similar transformation between the 3D points on marker coordinates and the correspondences on slam coordinate system. Finally, use this matrix to act on the camera pose to render the scene.
• My question is, how do I know which mappoints are in the current frame and the corresponding descriptor and also descriptor saved in mappoint seems to be the best descriptor of the associated keyframe? Can not be used to match the marker features directly? and how do I draw matches to verify whether the matches is right or not?
• Another problem is whether the GetTrackedMapPoint () function and GetTrackedKeyPointsUn () located in file named System.cc which return a vector of keypoint and a vector of mappoint respectively belongs to the current frame? is a keypoint at index i correspond to a mappoint at the same position? How to match the extracted feature in the marker with mappoint?
• A little more problems, Is there any problem with my approach? if no how to do it? Thanks in advance, any help will be appreciated.

[AlejandroSilvestri]

GetTrackedMapPoint() returns mTrackedMapPoints = mpTracker->mCurrentFrame.mvpMapPoints; GetTrackedKeyPointsUn() returns mTrackedKeyPointsUn = mpTracker->mCurrentFrame.mvKeysUn;

mpTracker->mCurrentFrame is a Frame, the current one.

mvpMapPoints is a vector of pointers to MapPoints, with the actual tracked ones.

Frame::mvpMapPoints, Frame::mvKeys, mvKeysUn and Frame::mDescriptors (among others) are aligned. All of them have Frame::N elements, corresponding to keypoints detected on actual image. The same index correspond to the same feature in all of this containers. Same index links a keypoint with its descriptor and its mappoint (if any, may be NULL).

How will you ensure there will be a mappoint where the marker is? Only mappoints have 3D location. (To be accurate, Frame and KeyFrame have it too, but I think we aren't use them here.)

[hechaochao]

@AlejandroSilvestri Thanks for your immediate reply. I have reviewed #309 what i want to do is the same problem, also have read the paper you suggested but not enlightened a lot, still have some problem in implementing the need.

• in order to judge which mappoints are belong to a marker is what i have did but encountered some problem. My idea is compare descriptors of keypoints of marker which has prepared before slam running, with the descriptors of keypoints returned by calling function GetTrackedKeyPointsUn(), the descriptors get from the corresponding mappoints, but according to the comment, this descriptor is the best fast descriptor, can i use it directly? or should i get the current frame then get the variable member named mDescriptors of class Frame?

• is this approach exist any problem? Hope to get your answer, thanks in advance.

[AlejandroSilvestri]

@hechaochao,

I am quite out of sync here, let me explain the basics perhaps you already know.

ORB uses FAST for feature detection, plus BRIEF for binary descriptor extraction. A keypoint is no much more than the pixel coordinates of the feature detected by FAST. A descriptor is a vector briefing the keypoints' surroundings. GetTrackedKeyPointsUn gives you keypoints, but you still need their descriptors, you can get from mDescriptors.

mDescriptors is a Mat, each row is a 256 bit long descriptor (4 32 bits ints I think, I don't remember exactly). You can use descriptors directly.

You can use

int ORB_SLAM2::ORBmatcher::DescriptorDistance(const cv::Mat & a, const cv::Mat & b)

to get the distance between two descriptors. The lower, the better.

[AlejandroSilvestri]

@hechaochao,

I'm a little out of sync here, let me begin from the basics, perhaps you already know.

ORB uses FAST as feature detector and BRIEF as descriptor extractor.

FAST over an image gives you a vector of keypoints. A keypoint isn't much more than the pixel coordinates of a feature. You still need its descriptor. You can get keypoints from GetTrackedKeyPointsUn() and descriptors from mDescriptors.

mDescriptors is a Mat. Each row is a 256 bits descriptor, packed in 4 32 bits int.

ORB_SLAM2::ORBmatcher::DescriptorDistance(...) gives you the distance between two descriptors. The lower, the better. In orb-slam2, distance goes from 0 to 255, because it uses 256 binary descriptors.

In this context there's no such thing as "marker descriptor". BRIEF descriptors are extracted from each keypoint' surroundings. Each descriptor describes a keypoint, the coordinates of a single image pixel.

May be you can identify a good and distinctive descriptor for a marker. May be you need more than one descriptor for a marker. If you end up using many descriptor for each marker, you should consider using BoW for marker recognition. I've no experience with this technology. I know how it works, but I don't know if it is ok for markers or objects. If I recall correctly, BoW was made for place recognition (the whole image).

[hechaochao]

@AlejandroSilvestri Thanks for your detail response. I write a piece of code to test the effect of extractoring orb feature and matching between two images, but the match is worse, almost all are mismatch i have used knnMatch to get good matches, but the result is not what i expected, is there any wrong exist in my code? when caculate the distance i also have tried call ORB_SLAM2::ORBmatcher::DescriptorDistance(...) but the result is same.

`ORBextractor mORBextractor = ORBextractor(1000,1.2,8,20,7);

cv::Mat img1_1 = cv::imread("timg.jpeg"); cv::Mat img2_1 = cv::imread("img2.jpg");

cv::Mat img1,img2; cvtColor(img1_1,img1,CV_BGR2GRAY); cvtColor(img2_1,img2,CV_BGR2GRAY);

vector< KeyPoint > keypoints1, keypoints2; Mat des1, des2; mORBextractor(img1, noArray(), keypoints1, des1); mORBextractor(img2, noArray(), keypoints2, des2);

Ptr matcher = DescriptorMatcher::create("BruteForce-Hamming"); vector< vector< DMatch > > matches; matcher->knnMatch(des1, des2, matches, 2); vector< DMatch > goodMatches; for(unsigned i = 0; i < matches.size(); i++) { if(matches[i][0].distance < 0.8f * matches[i][1].distance) { goodMatches.push_back(DMatch(matches[i][0].queryIdx, matches[i][0].trainIdx, matches[i][0].distance)); } }

Mat matchImg; drawMatches(img1, keypoints1, img2, keypoints2, goodMatches, matchImg); imshow("matchResult", matchImg); cv::waitKey(0);`

Below is the matching result

[AlejandroSilvestri]

@hechaochao ,

I believe your code is ok. You are looking for matches in two unrelated images, and on top of it, without a threshold to discard poor matches (high distance matches).

ORB-SLAM2 uses ORB to look for matches between two consecutive frames (very similar images) and applying a lot of filtering, one of them geometrical, avoiding brute force matching, thus avoiding matches between two unrelated places.

It also apply knnMatch with k=2, with a maximum distance threshold for the first match, and a minimum distance difference between the first and the second match, discarding both if they are two similar.

You can see a similar result to yours during orb-slam2 initialization step. Almost brute force matching is performed in initialization, orb-slam2 draws the matches on screen, you can see many mismatches going on.

[hechaochao]

@AlejandroSilvestri Thanks. I remember that when executing loop closing, will search the KeyFrameDatabase which presented by words using Bow. Under these condition, will there occur the parallex of two images are far?How did you deal with that situation?or in the search process which are compared with the current frame is all near closely with the current one? Bow is use for speeding up the matching process, is there any difference if i don't use that except considering the time complexity?

[AlejandroSilvestri]

@hechaochao ,

Yes, BoW is a way to speed up descriptor matching. It has another advantage: orb-slam2 already has a BoW database and methods to compare images, may be you want to use them to look for markers. But there is a lot to learn before using them.

About loop closing: it fuse two distant keyframes in one, and correct the whole map (mappoints and keyframe poses). You could use map correction methods to apply your markers known poses. Again, some learning required.

[mromanelli9]

Hi to you all. I'm trying to do a similar thing: using the ArUco markers (OpenCV) I would like to have a real-world scale and thus get the depth of the points (or at least a good estimation). Using the ArUCo markers I can easily identify them in the current frame and also get the pose and rotation (rvecs and tvecs in aruco::estimatePoseSingleMarkers); obviously I have also the real size of the markers (eg. 10cm). My question is: once I've identified them and matched them to MapPoints (tracked ones), how and where sould I update/adjust the system with the new values? Feel free to correct me if I've done some mistakes along the way, I'm not an expert in this field.

p.s.: I'm using the video version, os1 (https://github.com/AlejandroSilvestri/os1).