search

rpautrat / SuperPoint

Efficient neural feature detector and descriptor
MIT License
1.88k stars 416 forks source link

problem about superpoint evaluation #193

Open jiahuiLiuliu opened 3 years ago

jiahuiLiuliu commented 3 years ago

Hi，rpautrat, sorry to disturb you. When I evaluate your model using sp_v6 you pretrained, I can't get the same result with your github's, but I evaluated the magicLeap's model， I have got the same result with your github's. The detailed results are as follows：

Descriptors evaluation: illumination changes viewpoint changes SuperPoint(your sp_v6, SuperPoint(COCO)) 0.94 0.678 SuperPoint(your gitbub release) 0.965 0.712 SuperPoint(MagicLeap) 0.933 0.766

The config I use is superpoint-hpatches: data: name: 'patches_dataset' dataset: 'hpatches' # 'coco' 'hpatches' alteration: 'v' # 'all' 'i' 'v' preprocessing: resize: [480, 640] # False for coco model: name: 'super_point' data_format: 'channels_last' batch_size: 50 eval_batch_size: 50 learning_rate: 0.001 detection_threshold: 0.001 # 1/65 nms: 8 top_k: 1000 homography_adaptation: num: 0 aggregation: 'sum' filter_counts: 10 homographies: translation: true rotation: true scaling: true perspective: true scaling_amplitude: 0.1 perspective_amplitude_x: 0.2 perspective_amplitude_y: 0.2 patch_ratio: 0.85 max_angle: 1.57 allow_artifacts: false eval_iter: 600 seed: 1

This result confuse me a long time, so could you give me some suggestion? Thanks very much.

rpautrat commented 3 years ago

Hi,

I am not sure to understand what is the difference between the sp_v6 model and the "github release".

But otherwise, the homography adaptation metric is based on RANSAC to estimate a homography, so it is a stochastic process. It could explain why you would get different results between different runs. Increasing the number of RANSAC iterations may reduce this fluctuation though.

jiahuiLiuliu commented 3 years ago

Thanks for your reply.

"github release" is your test result, "sp_v6" is your pretrained model that I downloaded, then I evaluated it use: python export_descriptors.py configs/superpoint_hpatches.config sp_v6 --export_name=sp_v6-hpatches-v the difference between them is viewpoint changes， “github release” ：0.712 "sp_v6"：0.678.

and I will increase the number of RANSAC iterations for evaluation.

rpautrat commented 3 years ago

I see, I think that the github release as published on the Readme was computed from the sp_v6 model, if I am not mistaken. So the difference between the two could just be the RANSAC stochasticity. Especially as you seem to have different results with the Magic Leap model (0.923/0.742 in my case, 0.933/0.766 in your case), and this model is certainly the same between both evaluations.

jiahuiLiuliu commented 3 years ago

yes, you are right, github release means the result which published on the Readme. I have increased the number of RANSAC iterations for evaluation to 10000, but the result didn't change anything. I think the difference between the two results with the Magic Leap model is so small, that I can ignore it, but the result I evaluated with sp_v6 is obvious lower than your published. when I test checkerboard images, I found the number of keypoints is obvious less than magicLeap's model, I can't paste the image in the github, could I sent the detail result images to your email?

rpautrat commented 3 years ago

Well, the difference between the ReadMe SP of Magic Leap and yours seems quite similar as the difference between your sp_v6 and the Github release (0.024 vs 0.034 for viewpoint for example). But if increasing the RANSAC iterations didn't help, I am not sure where the difference could come from.

You can send me the images to remi.pautrat[at]gmail.com if you want. But I cannot guarantee to find the issue...

SuanNaiShuiGuoLao commented 3 years ago

yes, you are right, github release means the result which published on the Readme. I have increased the number of RANSAC iterations for evaluation to 10000, but the result didn't change anything. I think the difference between the two results with the Magic Leap model is so small, that I can ignore it, but the result I evaluated with sp_v6 is obvious lower than your published. when I test checkerboard images, I found the number of keypoints is obvious less than magicLeap's model, I can't paste the image in the github, could I sent the detail result images to your email?

Hi, I just encountered a same problem that the results I got from sp_v6 are not as good as what listed in the page. Have you solved the problem?