Missing RobotCar/queries

[Merical]

Hi~ I'm trying to evaluate my method on robotcar with your amazing image-matching-toolbox. I notice that there should be a queries folder under RobotCar dataset, which is missing from the dataset download page https://data.ciirc.cvut.cz/public/projects/2020VisualLocalization/RobotCar-Seasons/ . Would you please provide the download link of this folder?

[GrumpyZhou]

Hi @Merical ,

Thanks for reporting this issue. I have uploaded it and updated the docs.

[Merical]

Thank you for your response! BTW, would you please help to provide some advice on producing evaluation results of aachen-day-night with the standard pipeline on visuallocalization.net? I've found it struggling to follow the provided repo https://github.com/tsattler/visuallocalizationbenchmark .

[GrumpyZhou]

Hi @Merical ，

What you need to do is simply changing the setting in configs. Here's how I test Patch2Pix for night only benchmark.

https://github.com/GrumpyZhou/image-matching-toolbox/blob/76a865780919ae87cabc050b4569cf0feb11007d/configs/patch2pix.yml#L38-L47

[Merical]

Thank you! So I can test with or without hloc by setting covis_cluster to True or False accordingly, is that right?

[GrumpyZhou]

Hi @Merical ,

Hm, I am not sure what you mean by with and without HLoc. While local feature benchmark and HLoc are implemented slightly different especially in the query registration step, the overall workflow is the same. One main difference between them is that local feature benchmark only evaluates the night query and reconstructs the model using different number of db pairs. So to switch between HLoc full evaluation and local feature benchmark (night only), you simply change the setting of pairs for reconstruction and registration. Here shows you three configs to run on full evaluation on Aachen 1.0, Aachen 1.1 and Aachen night 1.0:

https://github.com/GrumpyZhou/image-matching-toolbox/blob/76a865780919ae87cabc050b4569cf0feb11007d/configs/patch2pix.yml#L18-L47

But notice, you need to adapt these settings for your own method.

[Merical]

Thank you for your response! I should have better described my question. I understand that the image-matching-toolbox is built with hloc (https://github.com/cvg/Hierarchical-Localization). And I've noticed that some benchmark results in recent papers are tested on aachenv1.1 or v1.0 (day and night) with the standard pipeline provided in the ECCV workshop, i.e. https://github.com/tsattler/visuallocalizationbenchmark/tree/master/local_feature_evaluation. I wonder if it is possible to switch the pipeline between hloc and the standard one in image-matching-toolbox? It would be really helpful.

[GrumpyZhou]

Oh I see. Now I understand your previous question about the covisibility cluster option. So currently immatch hasn't integrated the original local feature evaluation yet and I might consider to add that in the future. Back to your question whether changing pairs + covis_cluster=False achieves the switch.

Local feature benchmark use colmap _imageregistrator interface directly as shown in https://github.com/tsattler/visuallocalizationbenchmark/blob/1f8e4311e6fc41519e3d9909e14fce9a7f013732/local_feature_evaluation/reconstruction_pipeline_aachen_v1_1.py#L245-L257 . For Hloc it is re-implemented as shown in hloc/localize_sfm.py.
But there pycolmap is used for pose estimation, so I believe the two implementations are rather similar when using the same image pairs and setting covis_cluster=False. Maybe you can also manually compare original colmap code of _imageregistrator with how hloc was implemented.. I hope this answer is helpful.

[Merical]

Cool! Thank you~ I'm closing this issue.

[Master-cai]

@GrumpyZhou Sorry to bother you in a closed issue, I just want to check the meaning of the covis_cluster flag. I find that it is set to different values in the yml you provided for different methods. Could you please explain further?