MeshLoc

This repository contains the code for our publication MeshLoc: Mesh-Based Visual Localization, which is going to be presented as poster at ECCV 2022.

License

This repository is licensed under the 3-Clause BSD License. See the LICENSE file for full text.

Citation

If you are using the code in this repository, please cite the following paper:

@inproceedings{Panek2022ECCV,
  author = {Panek, Vojtech and Kukelova, Zuzana and Sattler, Torsten},
  title = {{MeshLoc: Mesh-Based Visual Localization}},
  booktitle={European Conference on Computer Vision (ECCV)},
  year = {2022},
}

Installation

follow Image Matching Toolbox installation guide
- clone the repository (git clone https://github.com/GrumpyZhou/image-matching-toolbox.git)
- prepare immatch conda environment (conda env create -f environment.yml)
- activate the immatch environment (conda activate immatch)
- install immatch toolbox as Python package (python setup.py develop)
  - possibly update of PyTorch package will be necessary (e.g. conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch)
install other necessary packages to immatch environment (conda install pyyaml)
clone the MeshLoc repository
- git clone --recurse-submodules https://github.com/tsattler/meshloc_release.git
install mesh_loc Python package to immatch environment
- cd <mesh_loc_dir>
- python setup.py develop

Usage

Current implementation contains localization scripts for two datasets - Aachen v1.1 and 12 Scenes. The scripts should be launched from image-matching-toolbox directory so the scripts have direct access to the feature matching configuration files. Also activate the prepared immatch conda environment with all necessary packages.

Example on Aachen v1.1 dataset:

cd <image_matching_toolbox_dir>

conda activate immatch

python3 <mesh_loc_dir>/localize.py \
--db_image_dir <meshloc_dataset_path>/aachen_day_night_v11/images/images_db_undist_800 \
--db_depth_image_dir <meshloc_dataset_path>/aachen_day_night_v11/db_renderings/AC14_depth_800_undist \
--colmap_model_dir <meshloc_dataset_path>/aachen_day_night_v11/db_colmap_models/800_undist \
--query_dir <meshloc_dataset_path>/aachen_day_night_v11/images/images_q_night_800 \
--query_list <meshloc_dataset_path>/aachen_day_night_v11/night_time_queries_with_intrinsics_800_basenames.txt \
--out_prefix <experiment_outputs_dir_path> \
--match_prefix <experiment_matches_dir_path> \
--method_name patch2pix \
--method_config aachen_v1_1 \
--method_string patch2pix_aachen_v1_1_ \
--retrieval_pairs <meshloc_dataset_path>/aachen_day_night_v11/retrieval_pairs/NetVLAD_top50_underscores.txt \
--top_k 50 \
--max_side_length -1 \
--ransac_type POSELIB+REF \
--min_ransac_iterations 10000 \
--max_ransac_iterations 100000 \
--reproj_error 20.0 \
--use_orig_db_images \
--cluster_keypoints

Example on 12 Scenes dataset:

cd <image_matching_toolbox_dir>

conda activate immatch

python3 <mesh_loc_dir>/localize_12scenes.py \
--db_image_dir <12_scenes_dataset_path>/office1/manolis/data \
--db_depth_image_dir <meshloc_dataset_path>/12_scenes/db_renderings/12_scenes_apt1_kitchen_depth \
--colmap_model_dir <meshloc_dataset_path>/12_scenes/db_colmap_models/apt1_kitchen \
--query_dir <12_scenes_dataset_path>/office1/manolis/data \
--query_list <meshloc_dataset_path>/12_scenes/queries_with_intrinsics/apt1_kitchen_queries_with_intrinsics.txt \
--out_prefix <experiment_outputs_dir_path> \
--match_prefix <experiment_matches_dir_path> \
--method_name loftr \
--method_config default \
--method_string loftr_default_ \
--retrieval_pairs <meshloc_dataset_path>/12_scenes/retrieval_pairs/apt1_kitchen_DVLAD_top20.txt \
--top_k 20 \
--max_side_length -1 \
--ransac_type POSELIB+REF \
--min_ransac_iterations 10000 \
--max_ransac_iterations 100000 \
--reproj_error 20.0 \
--use_orig_db_images

List of localization script arguments:

db_image_dir = directory with database images
db_depth_image_dir = directory with depth database images (assume _depth.npz postfix)
rendering_postfix = postfix of rendered database images if used (our IBMR rendering pipeline produces either _rendered_color.png or _rendered_no_color.png depending on shader preset)
colmap_model_dir = directory of a COLMAP model defining the database cameras - images file contains subpaths from db_image_dir to the images
query_dir = directory with query images
query_list = list of query images with intrinsics - contains subpaths from query_dir to the images
out_prefix = prefix (including path) of output file (with estimated query poses) in format accepted by The Visual Localization Benchmark
match_prefix = prefix (including path) for files containing local matches between query and database images
method_name = matching method name - see file names in image-matching-toolbox/configs - e.g. patch2pix
method_config = one of the method configurations - see configurations inside individual YAML files (e.g. image-matching-toolbox/configs/patch2pix.yml) - e.g. aachen_v1_1
method_string - string used for your identification of matching method in output files - e.g. patch2pix_aachen_v1_1_
retrieval_pairs = file from a retrieval step containing similar database images for each query image (text file, each line contains query_image_file database_image_file)
top_k = number of top retrieved database images, which will be used for localization
max_side_length = longer side of the database images - for scaling of the query intrinsics (-1 for full resolution)
ransac_type = RANSAC type (possible values: POSELIB, POSELIB+REF)
min_ransac_iterations = minimum number of RANSAC iterations
max_ransac_iterations = maximum number of RANSAC iterations
reproj_error = reprojection error RANSAC threshold
use_orig_db_images = set if using original images (not renderings) for db_image_dir
triangulate = set to use triangulation instead of 3D points from depth maps
merge_3D_points = set to select one of multiple 3D points available per query feature
cluster_keypoints = set to cluster keypoints (applicable only for patch2pix)
covisibility_filtering = set to use covisibility filtering
all_matches_ransac = use all possible 2D-3D matches in RANSAC
refinement_range = range for the +REF refinement
refinement_step = step size for the +REF refinement
bias_x = bias term for x-direction for feature detections
bias_y = bias term for y-direction for feature detections

The pose estimates from the localization pipeline on Aachen v1.1 can be evaluated by benchmark at https://www.visuallocalization.net/. The pose estimates on 12 Scenes can be evaluated by the https://github.com/tsattler/visloc_pseudo_gt_limitations repository, which also contains links to 12 Scenes COLMAP models.

Data

We used Aachen v1.1 and 12 Scenes datasets for the evaluation in our paper.

The data derived from the above mentioned datasets (such as meshes, renderings, or COLMAP models) can be found in our data repository at https://data.ciirc.cvut.cz/public/projects/2022MeshLoc.

You can use the download script to easily get the whole data repository.

Script parameters:

if no parameters are passed, the whole data repository will be downloaded to current directory
-n < all | aachen | 12_scenes > - specifies which dataset to download
-p \< string > - specifies the directory where the data will be downloaded
-z - unzips everything and removes the zip files

Acknowledgements

This repository is heavily using PoseLib, RansacLib and Image Matching Toolbox. We would like to thank all the contributors of these repositories.

tsattler / meshloc_release

readme