Get global_SFM and local_SFM results different from the listed benchmark results

[shuimushan]

I follow the Image Retrieval Benchmark tutorial but got global_SFM and local_SFM results different from the listed benchmark results, whereas the EWB results are the same with the given benchmark. I ran kapture_pipeline_image_retrieval_benchmark_from_pairsfile.py with COLMAP 3.8 ,pycolmap 0.1.0, and LFEAT=feather2d2_dim32_20k.

1. run full benchmark (all 3 tasks) for AP-GeM-LM18_top20 on Gangnam Station B2

Model: global_sfm_config_2

Found 911 / 916 image positions (99.45 %). Found 911 / 916 image rotations (99.45 %). Localized images: mean=(25.8816m, 36.4095 deg) / median=(0.0750m, 1.3386 deg) All: median=(0.0756m, 1.3410 deg) the listed results: median=(0.0756m, 1.2993 deg) Min: 0.0033m; 0.0537 deg Max: 994.0691m; 179.9981 deg

(0.25m, 2.0 deg): 56.00% the listed results: 35.59% (0.5m, 5.0 deg): 61.46% the listed results: 56.66% (5.0m, 10.0 deg): 63.21% the listed results: 62.23%

Model: local_sfm

Found 378 / 916 image positions (41.27 %). Found 378 / 916 image rotations (41.27 %). Localized images: mean=(23.2583m, 33.8776 deg) / median=(0.1011m, 1.5585 deg) All: median=(infm, inf deg) the listed results: median=(infm, inf deg) Min: 0.0063m; 0.1690 deg Max: 245.3013m; 179.9680 deg

(0.25m, 2.0 deg): 22.38% the listed results: 12.88% (0.5m, 5.0 deg): 25.55% the listed results: 21.18% (5.0m, 10.0 deg): 27.18% the listed results: 25.33%

Model: EWB

Found 916 / 916 image positions (100.00 %). Found 916 / 916 image rotations (100.00 %). Localized images: mean=(53.1739m, 75.3842 deg) / median=(51.6639m, 63.4245 deg) All: median=(51.6639m, 63.4245 deg) It is exactly the same with the listed results. Min: 1.1050m; 0.8699 deg Max: 161.1966m; 179.7832 deg

(0.25m, 2.0 deg): 0.00% (0.5m, 5.0 deg): 0.00% (5.0m, 10.0 deg): 0.00% It is exactly the same with the listed results.

2. run full benchmark (all 3 tasks) for fire_top20 on Gangnam Station B2

Model: global_sfm_config_2

Found 914 / 916 image positions (99.78 %). Found 914 / 916 image rotations (99.78 %). Localized images: mean=(19.5645m, 19.1694 deg) / median=(0.0507m, 0.9815 deg) All: median=(0.0508m, 0.9876 deg) the listed results: median=(0.0518m, 1.0021 deg) Min: 0.0053m; 0.0648 deg Max: 4749.6707m; 179.9998 deg

(0.25m, 2.0 deg): 69.98% the listed results: 47.27% (0.5m, 5.0 deg): 75.44% the listed results: 69.10% (5.0m, 10.0 deg): 77.07% the listed results: 75.22%

Model: local_sfm

Found 588 / 916 image positions (64.19 %). Found 588 / 916 image rotations (64.19 %). Localized images: mean=(14.1432m, 21.6597 deg) / median=(0.0693m, 1.2338 deg) All: median=(4.8720m, 4.5286 deg) the listed results: median=(5.1373m, 4.4642 deg) Min: 0.0041m; 0.1077 deg Max: 197.2665m; 179.9642 deg

(0.25m, 2.0 deg): 40.17% the listed results: 25.44% (0.5m, 5.0 deg): 46.18% the listed results: 40.61% (5.0m, 10.0 deg): 48.80% the listed results: 46.51%

Model: EWB

Found 916 / 916 image positions (100.00 %). Found 916 / 916 image rotations (100.00 %). Localized images: mean=(35.9011m, 55.0691 deg) / median=(32.6906m, 40.2351 deg) All: median=(32.6906m, 40.2351 deg) It is exactly the same with the listed results. Min: 0.1836m; 0.4056 deg Max: 163.5414m; 179.9981 deg

(0.25m, 2.0 deg): 0.00%
(0.5m, 5.0 deg): 0.00%
(5.0m, 10.0 deg): 1.20% It is exactly the same with the listed results.

So why does this difference occur?

[yocabon]

Hi, it's probably a typo in the headers of the benchmark_results.adoc table (it's correct in the plots). gangnam station b2 global sfm and local sfm are evaluated with bins

(0.1, 1.0), (0.25, 2) and (1, 5) and not the ones written.

So you get

global_sfm_config_2 (0.25m, 2.0 deg): 56.00% and listed is actually 56.66% local_sfm (0.25m, 2.0 deg): 22.38% and listed is actually 21.18%

...

[shuimushan]

Hi, it's probably a typo in the headers of the benchmark_results.adoc table (it's correct in the plots). gangnam station b2 global sfm and local sfm are evaluated with bins

(0.1, 1.0), (0.25, 2) and (1, 5) and not the ones written.

So you get

global_sfm_config_2 (0.25m, 2.0 deg): 56.00% and listed is actually 56.66% local_sfm (0.25m, 2.0 deg): 22.38% and listed is actually 21.18%

...

So that's it. But I still wonder why I got 56.00% instead of 56.66%?

[yocabon]

There is some randomness (ransac) in the geometric verification step when computing the matches and during the localization itself (pnp-ransac).

[shuimushan]

There is some randomness (ransac) in the geometric verification step when computing the matches and during the localization itself (pnp-ransac).

Thanks! I get it.