Closed pete-machine closed 6 years ago
dgtlmoon
commented
6 years ago I've seen similar results although I'm not sure if it's related, but essentially I see that the descriptors are matching very quickly and reliably, but like in your screenshot, it's not the right descriptors from the same area/layout.. @komrad36 ideas? :)
pete-machine
commented
6 years ago I should mention, that the reason I am not using the K2NN, is that I am using the "mask" functionality available for the opencv matcher.
Another side note. I have adjusted above code slightly. For some unknown reason cv::Mat cv_desc0_latch needs to be set to zero explicitly.
// Old initialization
//cv::Mat cv_desc0_latch(cv::Size(64,keypoints0.size()),CV_8U);
//cv::Mat cv_desc1_latch(cv::Size(64,keypoints1.size()),CV_8U);
// NEW initialization
cv::Mat cv_desc0_latch = cv::Mat::zeros(keypoints0.size(),64,CV_8UC1);
cv::Mat cv_desc1_latch = cv::Mat::zeros(keypoints1.size(),64,CV_8UC1);I finally managed to figure out the problem... There are two vectors containing keypoints.
The latch descriptor and the brute-force matching was actually working. What I failed to realize (for many hours of debugging) is that the latch function will delete keypoints close to the border - deleting keypoints in the Latch::KeyPoint vector. Matching is done for the Latch::KeyPoint vector. However, when I am using the opencv function to draw/visualize matches, I am using the old cv::KeyPoint vector and matched indices from the Latch::KeyPoint vector.
I have added some code to get a working example. I have added a function to remove keypoints close to the border to highlight the issue.
#include <bitset>
#include <chrono>
#include <iostream>
#include <opencv2/opencv.hpp>
#include <vector>
#include "LATCH.h"
#define VC_EXTRALEAN
#define WIN32_LEAN_AND_MEAN
using namespace std::chrono;
// Removes interest points that are to close to the border
void remove_outside( std::vector<cv::KeyPoint>& keypoints,const int width, const int height, const int border_value) {
keypoints.erase(std::remove_if(keypoints.begin(), keypoints.end(), [width, height,border_value](const cv::KeyPoint& kp) {return kp.pt.x <= border_value || kp.pt.y <= border_value || kp.pt.x >= width - border_value || kp.pt.y >= height - border_value; }), keypoints.end());
}
int main(int argc, char **argv) {
// ------------- Configuration ------------
constexpr int numkps = 500;
constexpr bool multithread = true;
constexpr char name0[] = "test.jpg";
//constexpr char name1[] = "000000_cropped.jpg";
// --------------------------------
// ------------- Image Read ------------
cv::Mat image = cv::imread(name0, CV_LOAD_IMAGE_GRAYSCALE);
cv::Rect roi0(50, 50, image.cols/3-50, image.rows/3-50);
cv::Rect roi1(0, 0, image.cols/3, image.rows/3);
cv::Mat image0 = image(roi0);
cv::Mat image1 = image(roi1);
if (!image0.data || !image1.data) {
std::cerr << "ERROR: failed to open image. Aborting." << std::endl;
return EXIT_FAILURE;
}
// --------------------------------
// ------------- Detection ----------------------------
std::cout << std::endl << "Detecting..." << std::endl;
int edge_treshold = 31; // 36 is not working correctly.
cv::Ptr<cv::ORB> orb = cv::ORB::create(numkps, 1.2f, 8, edge_treshold, 0, 2, cv::ORB::HARRIS_SCORE, 31, 20);
std::vector<cv::KeyPoint> keypoints0;
std::vector<cv::KeyPoint> keypoints1;
orb->detect(image0, keypoints0);
orb->detect(image1, keypoints1);
// -------------------------------
// ------------- LATCH feature descriptor ------------
// IMPORTANT LINES to later draw the correspondes correctly. The opencv draw
// function requires vector<cv::KeyPoint> and Latch uses a vector<KeyPoint> (defined in LATCH.h).
// What I failed to notice for many hours of debugging is that the LATCH function will automatically remove
// keypoints close to the border (36 pixels). This causes problem.
// This can be fixed by simply removing keypoints close to the border.
remove_outside(keypoints0,image0.cols, image0.rows,36);
remove_outside(keypoints1,image1.cols, image1.rows,36);
// IMPORTANT! Convertion between OpenCV [n_keypoints x 64](uint8_t) and the latch descriptor [n_keypoints x 8](uint64_t) format
// The opencv bruteforce matching requires the data to be in a cv::Mat format.
// A cv::Mat with dimenstions [n_keypoints x 64] of type char/uint8_t is initialized. (A single row correspond to a keypoints with a (64x8=) 512 bits descriptor).
cv::Mat cv_desc0_latch = cv::Mat::zeros(keypoints0.size(),64,CV_8UC1);
cv::Mat cv_desc1_latch = cv::Mat::zeros(keypoints1.size(),64,CV_8UC1);
// A pointer is created to point to the data of the cv::Mat.
uint64_t* desc0_latch = (uint64_t*)(cv_desc0_latch.data);
uint64_t* desc1_latch = (uint64_t*)(cv_desc1_latch.data);
std::vector<Latch::KeyPoint> kps0_latch;
std::vector<Latch::KeyPoint> kps1_latch;
for (auto&& kp : keypoints0) kps0_latch.emplace_back(kp.pt.x, kp.pt.y, kp.size, kp.angle * 3.14159265f / 180.0f);
for (auto&& kp : keypoints1) kps1_latch.emplace_back(kp.pt.x, kp.pt.y, kp.size, kp.angle * 3.14159265f / 180.0f);
std::cout << "Size: " << kps0_latch.size() << keypoints0.size() << std::endl;
LATCH<multithread>(image0.data, image0.cols, image0.rows, static_cast<int>(image0.step), kps0_latch, desc0_latch);
LATCH<multithread>(image1.data, image1.cols, image1.rows, static_cast<int>(image1.step), kps1_latch, desc1_latch);
// ------------- Matching ----------------------------
cv::BFMatcher matcher(cv::NORM_HAMMING,true);
std::vector< cv::DMatch > matches_latch;
matcher.match( cv_desc0_latch, cv_desc1_latch, matches_latch);
std::vector< cv::DMatch > good_matches_latch;
int threshold_latch = 50;
for(unsigned int i = 0; i < matches_latch.size(); i++ ) {
if( matches_latch[i].distance < threshold_latch) {
good_matches_latch.push_back( matches_latch[i]);
//std::cout << "Sample LATCH, queryIdx: " << matches_latch[i].queryIdx << ", trainIdx: " << matches_latch[i].trainIdx << ", Distance: " << matches_latch[i].distance << std::endl;
}
}
// TESTING ///////////////////
// To validate that keypoints are identical before and after the latch-function.
// I used many hours debugging, to notice that the latch-functions removes keypoints close than 36 pixels
// to the border.
/*for(unsigned int i = 0; i < keypoints0.size(); i++ ) {
std::cout << "KP Latc(before) : " << keypoints0[i].pt << std::endl;
std::cout << "KP Latc(before) : [" << kps0_latch[i].x << ", " << kps0_latch[i].y << "]" << std::endl << std::endl;
}*/
int print_keypoint_idx = 0; // keypoints_class.size()
std::cout << "argc:" << argc << std::endl;
if (argc > 1) {
std::cout << "Select keypoint0 " << argv[1] << "/" << keypoints0.size()-1 << std::endl;
std::cout << "Select keypoint1 " << argv[1] << "/" << keypoints1.size()-1 << std::endl;
print_keypoint_idx = std::atoi(argv[1]);
}
// LATCH ORIGINAL //////////////////////////////////////////////
std::cout << "Descriptor (LATCH original): " << print_keypoint_idx << "/" << keypoints0.size()-1 << std::endl;
int shift = print_keypoint_idx*8;
for (int i = 0; i < 8; ++i) {
std::cout << std::bitset<64>(desc0_latch[i+shift]) << std::endl;
}
std::cout << std::endl;
// LATCH CLASS ( FLIPPED!!!) /////////////////////////////////////////////////
std::cout << "Descriptor (LATCH cv_mat FLIPPED!!!): " << print_keypoint_idx << "/" << keypoints0.size()-1 << std::endl;
for (int i = 0; i < 8; ++i) {
for (int ii = 0; ii < 8; ++ii) {
std::cout << std::bitset<8>(cv_desc0_latch.at<unsigned char>(print_keypoint_idx,i*8+7-ii));
//std::cout << std::bitset<8>(cv_desc0_latch.at<uint8_t>(i*8+7-ii,print_keypoint_idx));
}
std::cout << std::endl;
}
std::cout << std::endl;
cv::Mat img_matches_latch;
cv::drawMatches( image0, keypoints0, image1, keypoints1,
good_matches_latch, img_matches_latch, cv::Scalar::all(-1), cv::Scalar::all(-1),
std::vector<char>(), cv::DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );
//-- Show detected matches
cv::namedWindow( "Good Matches (LATCH)", CV_WINDOW_NORMAL );
cv::imshow( "Good Matches (LATCH)", img_matches_latch );
std::cout << "LATCH: Descriptors size: " << cv_desc0_latch.size() << ", Good matches: " << good_matches_latch.size() << "(treshold " << threshold_latch << ")" << std::endl;
cv::waitKey(0);
}
komrad36
commented
6 years ago Hey Pete,
Sorry I didn't get back to you all earlier on this. Good find! Yes, OpenCV and I both remove edge keypoints, and it's entirely possible we do so differently. Likely, in fact.
-Kareem
First, thank you for providing so many high speed functions. I am trying to use your LATCH descriptor in combination with the bruteforce matcher of opencv. I am struggling to get good matches between two images using the latch descriptor. I have provided my test example below (which is an extended version of the main.cpp-function). The matching is done between the original and a cropped edition of the test.jpg image provided in your repo (the cropped image is automatically cropped in the provided code). In the first section of the code, I am able to successfully match the two images using ORB (see image below).
In the second section, I am failing to swap ORB with LATCH features (see image below).
Perhaps, I am doing something wrong? The most prone-to-error section is probably when the latch feature descriptor is converted from a uint64_t-type to the cv::Mat used in opencv. I initialize a cv::Mat to point to the uint64_t used by the latch code.
terminal output: