Train on our own dataset

[zxp771]

Hi @rpautrat Thanks for your great work and share. I have a problem with step 2) As we know the self-label process is found the repeatability of the detection. I'm wondering if we can mix the RGB dataset and infrared dataset to improve the heterogeneous image matching performance. I think if I can process the homographic adaption(in step 2) by myself maybe we can use the dataset(mixed the RGB and infrared images) and the H matrix between each image to label the pseudo-ground truth for further training.

[rpautrat]

Hi!

I think you can indeed mix RGB and infrared images together in the homographic adaptation. It should capture more details and allow the detector to find more general keypoints (instead of just corners visible in RGB) on RGB images.

To adapt step 2 to your own dataset, you need to create a parser for your dataset. You can copy the file datasets/coco.py (let's call it datasets/my_dataset) and modify it to adapt it to your own dataset. Then you just need to modify the config file configs/magic_point_coco_export.yaml (or create a new one for your own needs) and replace the line name: 'coco' by name: 'my_dataset'. You can then use this config file in step 2 and it should work.

[zxp771]

Hi @rpautrat Thanks for your suggestion. I think my poor English may not express my thought clearly. The purpose of my assumption is to find the general keypoints which will be detected on both RGB and infrared images. Then choose these points as the pseudo-ground truth. I think just mix the RGB and infrared images won't significantly improve the matching performance. What I want to express is we may need to instead the image during the homography adaption. Like the paper mentioned: I read your code but can't find anywhere to do this. Thx for your help again!

[rpautrat]

Sorry, I misunderstood you indeed. Yes, replacing the homographic warping (or adding besides the homographic warping) by the corresponding infrared image should help.

What you can do is to modify the file models/homographies.py. In the function homography_adaptation, you can either replace the tf.while_loop (if you want to completely remove the homographic warps) or add after the tf.while_loop a code that will add the detections of your infrared image to 'probs', a mask full of 1s to 'counts' and your infrared image to 'images'. Nothing else should be needed I think.

[zxp771]

Hi @rpautrat Thanks for your reply. This is really helpful for my work. Only one more question. As I know we need to compute the H matrix between the RGB image and the Infrared Image(We need one pair of the images' H at least.cause the Infrared image is added by myself.We can't get the H from the code(like sample_homography) directly, right? But It's easy to compute it.) This H will impact the corresponding for the different type of images. However, I don't know how to add it to the homographies.py. Thanks for your help and patience again!

[rpautrat]

You just need to multiply the homography matrices between them to compose them. For example for your image Infrared1, in the step function of homographies.py, you will need to:

• detect the points on Infrared1.
• multiply the H between Infrared1 and Infrared0 with the H between RGB0 and Infrared0 to get the transform between Infrared1 and RGB0.
• warp the detections in the frame of RGB0 using this transform.
• add the prob, count and image as explained in my previous comment.

[zxp771]

Hi @rpautrat Thanks for your reply and your suggestion. Yes, I know the corresponding of the two types of images is easy to build. My main question is to add the infrared images in the homography workflow and get the export detection. Here is the whole flow I suppose:(the red part is the work I should do) I still have some doubts 1)We only need one RGB image one Infrared image and one file contains the H matrix. Thus, when we read our datasets when can set one folder to save these three mentioned files. Here we need to modify the coco.py??? 2)We need to do the samply_homography process for the RGB and IR image Separately then detect the points on the data augmentation result(maybe 50 to 50 nums of different images). All of these processes will be done in the tf.while_loop or function "step"??? 3)We didn't find that the export_detection.py use the homographies so how it exports the labels?

[rpautrat]

1) You need to create a file similar to coco.py, but adapted to your dataset. You should for example arrange your dataset with multiple folders and in each of them the RGB image, the infrared one and the homography between the two images. The new file would then read all these datasets and store their paths. One sample of your custom dataset would then be a dictionary wiht 3 entries: the RGB, the infrared and the homography. You can also imitate the file patches_dataset.py that creates another custom dataset.

2) I think that you need to sample from the dataset created in 1), and for each sample use the 'step' function to do what you want. In the 'step', you would then sample homographies (for example 50 for the RGB and 50 for the infrared, maybe with a random variable with a 0.5 proba to choose one or the other) and do the detections on each as it is currently done. It is not so hard to implement I think, so just try it, I won't write it for you!

3) The function homography_adaptation of homographies.py is used in magic_point.py, line 36. A simple global search in a folder (which is available in any recent code editor) would have told you that!

[zxp771]

Hi @rpautrat Thanks for your reply and suggestion! I'll close this issue. Thanks for your help again.

[zxp771]

Hi @rpautrat Sorry to disturb again. I was confused by the parameter count/counts. I remember you said 'counts' means 'a mask full of 1s' but I still don't know how it works in homographies.py.

[rpautrat]

Counts is an array containing all the count variables. The first count is initialized full of 1s because it corresponds to the original (unwarped) image. The following count variables store a mask that hides the border artifacts created by the warping.

[zxp771]

Hi @rpautrat Thx for your help. BTW you said "add the prob, count, and image".Do you mean just use the tf.concat to add these parameter together?(such as prob = tf.concat([prob_first_ir,prob_rgb,prob_ir]))

[rpautrat]

Yes, something like this.

[zxp771]

Hi @rpautrat Thanks for your help and patience.

[zxp771]

Hi @rpautrat Thanks for your help, now I got the label as we wished. But I got a problem when I trained the super-point detector. The training process is normal before the 65000 iterations. after the "RuntimeWarning" we lose the precision for several times The precision showed "nan“ when the iteration is larger than 65000. I'm wondering if it caused but our dataset is too small for such iteration number. we only use 300 rgb/ir images for training the super point.

[09/21/2019 09:31:40 INFO] Iter 0: loss 30.1992, precision 0.0001, recall 0.0046 [09/21/2019 09:49:53 INFO] Saving checkpoint for iteration #5000 [09/21/2019 09:50:18 INFO] Iter 5000: loss 6.1249, precision 0.0004, recall 0.0252 [09/21/2019 10:08:34 INFO] Saving checkpoint for iteration #10000 [09/21/2019 10:08:52 INFO] Iter 10000: loss 3.9210, precision 0.0006, recall 0.0423 [09/21/2019 10:27:09 INFO] Saving checkpoint for iteration #15000 [09/21/2019 10:27:11 INFO] Iter 15000: loss 2.3458, precision 0.0005, recall 0.0245 [09/21/2019 10:45:29 INFO] Saving checkpoint for iteration #20000 [09/21/2019 10:45:30 INFO] Iter 20000: loss 1.4084, precision 0.0091, recall 0.0162 [09/21/2019 11:03:48 INFO] Saving checkpoint for iteration #25000 [09/21/2019 11:03:50 INFO] Iter 25000: loss 0.8128, precision 0.0139, recall 0.0204 [09/21/2019 11:22:08 INFO] Saving checkpoint for iteration #30000 [09/21/2019 11:22:10 INFO] Iter 30000: loss 0.4833, precision 0.0303, recall 0.0285 [09/21/2019 11:40:28 INFO] Saving checkpoint for iteration #35000 [09/21/2019 11:40:29 INFO] Iter 35000: loss 0.3724, precision 0.0384, recall 0.0393 [09/21/2019 11:58:47 INFO] Saving checkpoint for iteration #40000 [09/21/2019 11:58:48 INFO] Iter 40000: loss 0.3154, precision 0.0719, recall 0.0284 [09/21/2019 12:17:06 INFO] Saving checkpoint for iteration #45000 [09/21/2019 12:17:08 INFO] Iter 45000: loss 0.2602, precision 0.0434, recall 0.0204 [09/21/2019 12:35:25 INFO] Saving checkpoint for iteration #50000 [09/21/2019 12:35:27 INFO] Iter 50000: loss 0.1730, precision 0.0521, recall 0.0298 [09/21/2019 12:53:44 INFO] Saving checkpoint for iteration #55000 [09/21/2019 12:53:45 INFO] Iter 55000: loss 0.2102, precision 0.0000, recall 0.0165 [09/21/2019 13:12:04 INFO] Saving checkpoint for iteration #60000 [09/21/2019 13:12:05 INFO] Iter 60000: loss 0.1343, precision 0.0147, recall 0.0226 [09/21/2019 13:30:23 INFO] Saving checkpoint for iteration #65000 /home/cv-group/SuperPoint-master/superpoint/models/base_model.py:391: RuntimeWarning: Mean of empty slice metrics = {m: np.nanmean(metrics[m], axis=0) for m in metrics} [09/21/2019 13:30:24 INFO] Iter 65000: loss 0.1302, precision nan, recall 0.0201 [09/21/2019 13:48:43 INFO] Saving checkpoint for iteration #70000 [09/21/2019 13:48:44 INFO] Iter 70000: loss 0.1301, precision nan, recall 0.0163 [09/21/2019 14:07:02 INFO] Saving checkpoint for iteration #75000 [09/21/2019 14:07:04 INFO] Iter 75000: loss 0.1118, precision 0.2062, recall 0.0468 [09/21/2019 14:25:22 INFO] Saving checkpoint for iteration #80000 [09/21/2019 14:25:23 INFO] Iter 80000: loss 0.1564, precision nan, recall 0.0214 [09/21/2019 14:43:41 INFO] Saving checkpoint for iteration #85000 [09/21/2019 14:43:43 INFO] Iter 85000: loss 0.1216, precision 0.2083, recall 0.0236 [09/21/2019 15:02:01 INFO] Saving checkpoint for iteration #90000 [09/21/2019 15:02:03 INFO] Iter 90000: loss 0.2042, precision nan, recall 0.0031

[rpautrat]

Hi, A dataset of 300 images is definitely too small to train a deep learning system like SuperPoint, you need at least several thousands, or better, tens of thousands of images. The nan is probably due to the fact that your validation set is empty, probably because the original dataset was too small. You should probably check the size of your validation set.

[zxp771]

Hi @rpautrat Thanks for your reply. We are working on getting more images for training now. but it's really hard to get the corresponding RGB and IR images for tens of thousands. maybe several thousand is possible for our drone. Thanks for your sharing and help.

[zxp771]

Hi @rpautrat

we collected some new data. now we have 1000 RGB/IR images. I remembered that you said we need to modify the size of the validation set. Do you mean we should modify the parameter "validation_size" in superpoint_coco.yaml? The original value is 96. I think it's for the COCO dataset(which have more than 20000 images) could you give me some suggestions for setting this value for our dataset?

Thanks for your help!

[rpautrat]

I would use a validation set of size 50 to 100 in that case (so 5-10% of the full dataset).

If you still get Nan with this parameter, check that your dataset file correctly created the validation set by printing the size of the validation step at each validation step.

[zxp771]

Hi @rpautrat it work! thanks for your help!

[zxp771]

Hi @rpautrat sorry to disturb you again. Now We got the trained model(ckpt files). We hope to compare the superpoint result with other methods for different resource images.(use the cv::findhomogrphy) like this fig: we write the C++ project to test the superpoint and SIFT. but never succeed when we tried to use opencv to load our trained model(we have converted the ckpt into pb/pbtxt files).

[rpautrat]

Hi,

Is there a reason to do it in C++ necessarily? Because you can also use the evaluation files provided in this repo to compare SuperPoint (that you can then use with your ckpt file) with other models (like SIFT).

Among the metrics implemented in this repo, there is in particular the homography estimation evaluation that you are using (in https://github.com/rpautrat/SuperPoint/blob/master/superpoint/evaluations/descriptor_evaluation.py).

[zxp771]

Hi @rpautrat For now, it not necessary but we plan to compare this method with other learning-based methods(may train by caffe/torch). But after we tried last week it seems that we may need much more time for this work. I also notice the "match_feature_demo.py".I think it's useful for our comparison work. Thanks for your suggestion! I will read the code you share carefully.

[zxp771]

Hi @rpautrat I tried our trained model on the "match_feature_demo.py" I used the "superpoint_coco.yaml" and "export model" to get the saved_model.pb file. I used two images (1 RGB 1IR)in our train dataset but the model didn't detect any points for matching. The train log is here( I think it's fine): [10/15/2019 19:15:24 INFO] Iter 45000: loss 0.2271, precision nan, recall 0.0009 [10/15/2019 19:34:04 INFO] Iter 50000: loss 0.3168, precision 0.0000, recall 0.0019 [10/15/2019 19:52:44 INFO] Iter 55000: loss 0.2406, precision 0.0500, recall 0.0030 [10/15/2019 20:11:20 INFO] Iter 60000: loss 0.2789, precision 0.1667, recall 0.0067 [10/15/2019 20:30:01 INFO] Iter 65000: loss 0.3547, precision 0.2802, recall 0.0155 [10/15/2019 20:48:35 INFO] Iter 70000: loss 0.1414, precision 0.2800, recall 0.0155 [10/15/2019 21:07:15 INFO] Iter 75000: loss 0.1884, precision 0.1753, recall 0.0169 [10/15/2019 21:25:51 INFO] Iter 80000: loss 0.1749, precision 0.1349, recall 0.0157 [10/15/2019 21:44:25 INFO] Iter 85000: loss 0.1259, precision 0.2598, recall 0.0168 [10/15/2019 22:03:00 INFO] Iter 90000: loss 0.1927, precision 0.2311, recall 0.0191 [10/15/2019 22:21:34 INFO] Iter 95000: loss 0.1346, precision 0.3210, recall 0.0208 [10/15/2019 22:40:15 INFO] Iter 100000: loss 0.1497, precision 0.3878, recall 0.0219 [10/15/2019 22:58:50 INFO] Iter 105000: loss 0.1580, precision 0.1666, recall 0.0303 [10/15/2019 23:17:30 INFO] Iter 110000: loss 0.1681, precision 0.2509, recall 0.0235 [10/15/2019 23:36:05 INFO] Iter 115000: loss 0.2618, precision 0.2381, recall 0.0189 [10/15/2019 23:54:45 INFO] Iter 120000: loss 0.1406, precision 0.1571, recall 0.0103 [10/16/2019 00:13:25 INFO] Iter 125000: loss 0.1589, precision 0.1903, recall 0.0086 [10/16/2019 00:32:00 INFO] Iter 130000: loss 0.1859, precision 0.2925, recall 0.0159 [10/16/2019 00:50:36 INFO] Iter 135000: loss 0.1326, precision 0.3644, recall 0.0171 [10/16/2019 01:09:12 INFO] Iter 140000: loss 0.1372, precision 0.3069, recall 0.0135 [10/16/2019 01:27:53 INFO] Iter 145000: loss 0.1167, precision 0.2223, recall 0.0166 [10/16/2019 01:46:29 INFO] Iter 150000: loss 0.1861, precision 0.3119, recall 0.0181 [10/16/2019 02:05:02 INFO] Iter 155000: loss 0.1650, precision 0.2604, recall 0.0197 [10/16/2019 02:23:37 INFO] Iter 160000: loss 0.1901, precision 0.3929, recall 0.0183 [10/16/2019 02:42:17 INFO] Iter 165000: loss 0.1601, precision 0.4429, recall 0.0157 [10/16/2019 03:00:56 INFO] Training finished [10/16/2019 03:00:56 INFO] Saving checkpoint for iteration #170000

[rpautrat]

Hi,

Have you evaluated your detector (with repeatability for example) and descriptor (with homography estimation)? I mean with both kind of images.

[zxp771]

Hi @rpautrat Thanks for your reply! Yes, we tested our detector on RGB and IR images several hours ago. we train the magic-point(step2 and step3 like you said in Readme) for two times and test the repeatability. The results showed that the detected points decreased after every time we trained the magic-point(step2 and step3) and the super-point result has no detected point(or just have one point). For now, I think It may cause by the dataset is not enough but it's hard to expand it quickly(like coco dataset which has 20000 more images). I have two assumptions to improve our result:

1. I remember the "homography_adaption: num " in the magic-point_coco_export.yaml. Could we enlarge the value of it to improve the performance of our model since we can regard the homography adaption as the method for data propagation?
2. I also notice the "detection_threshold" Could we reduce the value of it for self-labeling (the step2) so we can get more detectors (maybe loss some accuracy.) to improve our model.

[rpautrat]

If there is no detection at all with SuperPoint while having good detection with MagicPoint, I think that it is more a problem with the implementation. Because there is no big difference for the detector in both cases (the encoder may still be modified by the descriptor branch in SuperPoint, but it still also optimized for detection).

1. You can try to increase the 'num' parameter when doing homography adaptation, but this will slow down the process. The original authors ( and I found similar results) showed that num = 1000 increases slightly the performance but is much slower compared to num = 100.

2. You can certainly tune the 'detection_threshold' parameter as well, since your case is now quite different from the original one.

[zxp771]

Hi @rpautrat Thanks for your reply. Yes, maybe the different resource images have few repeatability points that can be detected originally. Our implementation is quite special. I will try to modify the model by change these two parameters first. If it doesn't work I think I have no choice except expanding the dataset. Anyhow, thanks for your great work and suggestions for our work!

[zxp771]

Hi @rpautrat Sorry for disturbing time after time. We tried our trained model but I think we still have some problems. our trained model can detect and match some points(more than the model only trained on MS-COCO ) but the points' distribution on the corners are strange: The points we got are gathered very close.

We also tried the pre-trained model provided by @mmmfarrell. The distribution of detection points looks better(normal?) than our model.

I think this problem is impacted by the loss. for now, our best shot is loss:0.8496 precision:0.2200(We tried the learning rate decay function.) I have no idea to modify the result. Could you give us some advice? Can we modify the loss parameter "lambda_loss" in the config file?

[rpautrat]

Hi,

I think that the predictions clustered around a single corner points are due to a bug in the homographic adaptation. You can find a fix in this commit of branch superpoint_v1: 2efe4bc5a47fbffa594948b1d5c4e8af7ad3d34c Try to add this fix to your repo and to generate again the pseudo groundtruth with the corrected version, it should hopefully solve the problem.

[zxp771]

Hi @rpautrat Thanks for your advice. We tried your method today. 2019-11-05 08:34:24.233459: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x273269e0 [11/05/2019 08:34:27 INFO] Iter 0: loss 28.8081, precision 0.0028, recall 0.0072 [11/05/2019 08:52:31 INFO] Iter 5000: loss 1.6488, precision 0.0963, recall 0.1484 [11/05/2019 09:10:39 INFO] Iter 10000: loss 1.6573, precision 0.1437, recall 0.2282 [11/05/2019 09:28:45 INFO] Iter 15000: loss 1.1413, precision 0.1834, recall 0.3009 [11/05/2019 09:46:52 INFO] Iter 20000: loss 0.8701, precision 0.2322, recall 0.3419 [11/05/2019 10:04:58 INFO] Iter 25000: loss 0.6369, precision 0.2198, recall 0.3454 [11/05/2019 10:23:06 INFO] Iter 30000: loss 0.9390, precision 0.2056, recall 0.3269 [11/05/2019 10:41:14 INFO] Iter 35000: loss 0.5735, precision 0.2075, recall 0.3383 [11/05/2019 10:59:22 INFO] Iter 40000: loss 0.8946, precision 0.2259, recall 0.3482 [11/05/2019 11:17:30 INFO] Iter 45000: loss 0.6519, precision 0.2240, recall 0.3479 [11/05/2019 11:35:38 INFO] Iter 50000: loss 0.7400, precision 0.2188, recall 0.3489 [11/05/2019 11:53:47 INFO] Iter 55000: loss 0.7675, precision 0.2208, recall 0.3459 [11/05/2019 12:11:55 INFO] Iter 60000: loss 0.8865, precision 0.2191, recall 0.3452 [11/05/2019 12:30:03 INFO] Iter 65000: loss 1.0342, precision 0.2207, recall 0.3440 [11/05/2019 12:48:11 INFO] Iter 70000: loss 0.6299, precision 0.2170, recall 0.3463 [11/05/2019 13:06:19 INFO] Iter 75000: loss 0.8731, precision 0.2226, recall 0.3466 [11/05/2019 13:24:28 INFO] Iter 80000: loss 0.6383, precision 0.2197, recall 0.3445 [11/05/2019 13:42:36 INFO] Iter 85000: loss 0.5897, precision 0.2188, recall 0.3474 [11/05/2019 14:00:45 INFO] Iter 90000: loss 0.7561, precision 0.2217, recall 0.3462 [11/05/2019 14:18:53 INFO] Iter 95000: loss 0.7713, precision 0.2227, recall 0.3447 [11/05/2019 14:36:59 INFO] Training finished [11/05/2019 14:37:00 INFO] Saving checkpoint for iteration #100000 the result still have problem.(predictions clustered around a single corner) We used our pre-trained model(On MS_COCO as your introduction few monthes age) For now, I think our training on MS_COCO is not good enough as yours. So when we tried to do the transfer learning the result can't reach our target. Could you share your pre-train model with me(ckpt files and log file)? I think maybe we can do transfer learning with your pre-trained model labeled our dataset. Then training the model.

[rpautrat]

You based your code on the current master branch (that didn't change much recently), right? I can send you the ckpt of my training with this branch, but otherwise there is also a newer branch (superpoint_v1), with some recent improvements. Which model should I send you?

[zxp771]

Hi @rpautrat Thanks for your reply. Yes, we modified the current master branch for our implementation. I also modified the homography.py as your advice. I think your newer branch may have more improvement. but I don't know if the newer branch code is available for sharing. So I think the newer trained model(.ckpt files) is better. Thanks for your help and sharing.

[zxp771]

Hi @rpautrat I re-trained the model on MS_COCO. (I modified the homography.py as your advice) I use the repeatability code test the magic-point(trained on MS_COCO for two times) The distribution seems better than the version we trained before but we got a new problem. for the last magic-point model, the loss converge into 1.XXXX and the precision converge into 0.09XX I use this result to train the super-point model The loss can't conerge lower than 3.XXXX Could you give me some advice to solve this problem?

[rpautrat]

In MagicPoint, there is only one loss: the detector loss. In SuperPoint, there are first 2 detector losses (because it is trained on pairs of images), plus the descriptors loss. So it makes sense that the loss during training of SuperPoint is at least twice the loss in MagicPoint, and probably a bit more to account for the descriptors loss.

You can check in Tensorboard the value of each loss separately to verify that you get roughly the same loss as in the training of MagicPoint.

[zxp771]

Hi @rpautrat Thanks for your reply. Do you mean that the evaluation of loss and precision won't bring much impact on feature matching? BTW could you share the ckpt files of your pre-train model?

[rpautrat]

I didn't understand what you meant by "the evaluation of loss and precision won't bring much impact on feature matching". Precision is an observation variable to see if your network learns something meaningful and the loss is optimized to have reliable features and descriptors for matching. So of course the loss impacts feature matching.

You can get the ckpt file of the current superpoint_v1 branch here.

[zxp771]

Hi @rpautrat Thanks for your sharing. I mean the loss and precision value we got. I think it may a little larger(than our trained model before.) I'm afraid the large loss will bring us a bad performance of matching results. Thanks for your help.

[rpautrat]

But are you comparing the same losses? SuperPoint loss with SuperPoint loss? Or detector loss with detector loss?

[zxp771]

Hi @rpautrat Yes, I stored all of the results. I mean I kept the ckpt files for every time we trained on the MS_COCO dataset. (magic-point and super-point) And I compare the detector loss with the new detector loss the super-point loss with the new super-point loss.

[rpautrat]

Well, then a high loss is probably a bad sign, even though you still need to evaluate with a proper metric to compare the results. But if the difference in the loss is small and you had good result with the previous model, it should still be fine with the new model.

[zxp771]

Hi @rpautrat Thanks for your help. I will compare the model you send to me with our new model after we do the transfer learning on the RGB&IR images. Thanks for your advice and sharing.

[zxp771]

Hi @rpautrat Thanks for your share of sp_v6 But I think you may send me a wrong version. I tested your model on hpatches-i. It looks strang. Here is the screenshot.

[zxp771]

Hi @rpautrat I don't know what's wrong with the github. I can't upload the screenshot. could you give me your email?

[rpautrat]

For some reason, the screenshot appeared in the email sent by Git, but not online. Anyway, I saw your screenshot, which is indeed not what is expected. Are you sure you used the most recent version of branch superpoint_v1 to run the checkpoint?

[zxp771]

Hi @rpautrat I saw the superpoint_v1 is updated in April. so I think we are using the recent version. I remember we download your project in May.

[rpautrat]

You should pull superpoint_v1 then, I pushed new content yesterday on this branch.

[zxp771]

Hi @rpautrat I find it! Thanks for sharing! I will test the model you sent to me on this new version.

[zxp771]

Hi @rpautrat Thanks for sharing! I tested the model on your new "detector_repeatability_hpatches" it looks good now. It's really a great work! I also tried to export the model and loaded it for test the "match_feature_demo.py" but the result still strange. Is this caused by the model structure has been different from the old version? I use the "Netron" to open the model we trained before and yours sp_v6. I found the nodes(tensor name) are same.(superpoint/image,superpoint/prob_nms,superpoint/descriptors) I don't why the "match_feature_demo.py" got such strang result. Should I modified somewhere of the "match_feature_demo.py"?

[rpautrat]

Yes the model structure changed since the release of match_feature_demo.py (for example the descriptors are now bilinearly interpolated instead of using bicubic interpolation before).

So you re-exported the new sp_v6 model with export_model.py, right?

The problem is probably that the new model sp_v6 has been trained on normalized images between 0 and 1, whereas they were between 0 and 255 before. So try to normalize the images in match_feature_demo.py and it should hopefully solve the problem.

[paalpri]

@zxp771 is it possible to share your RGB-thermal image database ? I am also doing some related work and need RGB-Thermal dataset. I will collect some on my own to, will send you this when i get it if its of interest to you.