search

tsattler / visuallocalizationbenchmark

342 stars 58 forks source link

0/0/0 for day query in website #53

Closed Guteentak closed 3 years ago

Guteentak commented 3 years ago

Hello, thank you for revealing a good benchmark.

Is it a normal result to have zero results for day query when submitting results to the benchmark site? It is currently benchmarking using SIFT, and both version 1 and 1.1 of Aachen dataset produce the same results.

tsattler commented 3 years ago

Did you visually inspect the poses or did you verify in some other way whether the estimated poses are good? If yes, can you check whether the format of the text file that you are submitting matches the one the server expects?

Guteentak commented 3 years ago

Yes, I checked it through COLMAP gui.

For the text file I submitted, I checked that all test cases consist of only night query cases. For example, it can be seen that it comes out as 'IMG_20161227_173116.jpg 0.0688424 0.83849 -0.059498 -0.537267 -336.085 120.708 651.952'.

tsattler commented 3 years ago

You mean you are only submitting poses for nighttime images?

Guteentak commented 3 years ago

python reconstruction_pipeline_v1_1.py --dataset_path /local/aachen --colmap_path /local/colmap/build/src/exe --method_name sift

Yes, as a result of performing the above code, only the poses of night-time images are estimated. When I checked the provided database_v1_1.db, the images of day-time query were not included, is it affected?

tsattler commented 3 years ago

The code provided is for the local feature challenge held at past workshops, which only focuses on the nighttime images. Since no daytime images are included in the submission to the server, reporting 0/0/0 for daytime images is meaningful and expected behavior.