search

zju3dv / DetectorFreeSfM

Code for "Detector-Free Structure from Motion", CVPR 2024
Apache License 2.0
598 stars 34 forks source link

Issue running on custom data #50

Closed shankar-anantak closed 3 months ago

shankar-anantak commented 3 months ago

Hi,

I am having an issue running the pipeline on my custom image data.

$ python eval_dataset.py +demo=dfsfm.yaml

I setup the environment exactly as outlined in the Install.md, including COLMAP from hxy-123.

When i run the demo, however, 

024-07-03 20:20:54.714 | INFO     | src.dataset.coarse_sfm_refinement_dataset:__init__:98 - Load colmap results finish!                                                                                       | 1/2 [01:05<01:05, 65.93s/it]
2024-07-03 20:20:54.715 | INFO     | src.dataset.coarse_sfm_refinement_dataset:__init__:110 - Building keyframes begin....
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 21078/21078 [00:03<00:00, 6403.53it/s]
2024-07-03 20:20:58.030 | INFO     | src.post_optimization.post_optimization:post_optimization:133 - Scene data construct finish!
2024-07-03 20:20:58.030 | INFO     | src.post_optimization.post_optimization:post_optimization:150 - Multi-view refinement matching begin!████████████████████████████████████████████████████████▍ | 20902/21078 [00:03<00:00, 7668.37it/s]
[2024-07-03 20:20:58,061][pytorch_lightning.utilities.seed][INFO] - Global seed set to 666
/home/dev/miniconda3/envs/detectorfreesfm/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/dev/miniconda3/envs/detectorfreesfm/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
[2024-07-03 20:21:00,150][src.MultiviewMatcher.backbone.S2DNet.s2dnet][INFO] - Loading S2DNet checkpoint at /home/dev/A/Pipelines/DetectorFreeSfM/src/MultiviewMatcher/backbone/S2DNet/checkpoints/s2dnet_weights.pth.
2024-07-03 20:21:00.259 | INFO     | src.post_optimization.matcher_model.multiview_match_worker:build_model:41 - Load model from path: ['weight/multiview_matcher.ckpt']
2024-07-03 20:21:00.262 | INFO     | src.post_optimization.data_construct.construct_matching_data:__init__:194 - Assign bags begin!
2024-07-03 20:21:01.415 | INFO     | src.post_optimization.data_construct.construct_matching_data:__init__:197 - Assign bags takes: 1.153486728668213
2024-07-03 20:21:01.425 | INFO     | src.post_optimization.data_construct.construct_matching_data:__init__:200 - Chunk bags takes: 0.009866952896118164
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:39<00:00,  1.20s/it]
2024-07-03 20:21:41.021 | INFO     | src.post_optimization.matcher_model.multiview_match:multiview_matcher:64 - Matcher finish!
[2024-07-03 20:21:45,301][root][INFO] - Running the bundle adjuster.█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉      | 32/33 [00:39<00:00,  2.63it/s]
[2024-07-03 20:21:45,301][root][INFO] - colmap incremental_model_refiner --input_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/model --output_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/colmap_refined --database_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/database.db --image_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_images --Mapper.filter_max_reproj_error 2 --Mapper.tri_merge_max_reproj_error 2 --Mapper.tri_complete_max_reproj_error 2 --Mapper.extract_colors 1 --image_list_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/fixed_images.txt --Mapper.num_threads 16
E0703 20:21:45.384716 71289 colmap.cc:158] Command `incremental_model_refiner` not recognized. To list the available commands, run `colmap help`.
[2024-07-03 20:21:45,387][root][WARNING] - Problem with run_incremental_model_refiner for /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/model, existing.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:57<00:00, 58.59s/it]
2024-07-03 20:21:45.473 | INFO     | __main__:eval_core:81 - Finish Processing /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene.█████████████████████████████████████████████████████████| 2/2 [01:57<00:00, 57.30s/it]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [14:18<00:00, 858.24s/it]
2024-07-03 20:21:45.475 | INFO     | __main__:eval_core:84 - Worker 0 finish!

It looks like there is an issue occurring right after:

2024-07-03 20:21:41.021 | INFO     | src.post_optimization.matcher_model.multiview_match:multiview_matcher:64 - Matcher finish!
[2024-07-03 20:21:45,301][root][INFO] - colmap incremental_model_refiner --input_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/model --output_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/colmap_refined --database_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/database.db --image_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_images --Mapper.filter_max_reproj_error 2 --Mapper.tri_merge_max_reproj_error 2 --Mapper.tri_complete_max_reproj_error 2 --Mapper.extract_colors 1 --image_list_path /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/fixed_images.txt --Mapper.num_threads 16
E0703 20:21:45.384716 71289 colmap.cc:158] Command `incremental_model_refiner` not recognized. To list the available commands, run `colmap help`.
[2024-07-03 20:21:45,387][root][WARNING] - Problem with run_incremental_model_refiner for /home/dev/A/Pipelines/DetectorFreeSfM/SfM_dataset/ontario/scene/DetectorFreeSfM_loftr_official_coarse_only__scratch_no_intrin/temp_refined_kpts/model, existing.

I dont see incremental_model_refiner under 'colmap help' output


Example usage:
  colmap help [ -h, --help ]
  colmap gui
  colmap gui -h [ --help ]
  colmap automatic_reconstructor -h [ --help ]
  colmap automatic_reconstructor --image_path IMAGES --workspace_path WORKSPACE
  colmap feature_extractor --image_path IMAGES --database_path DATABASE
  colmap exhaustive_matcher --database_path DATABASE
  colmap mapper --image_path IMAGES --database_path DATABASE --output_path MODEL
  ...

Available commands:
  help
  gui
  automatic_reconstructor
  bundle_adjuster
  color_extractor
  database_cleaner
  database_creator
  database_merger
  delaunay_mesher
  exhaustive_matcher
  feature_extractor
  feature_importer
  hierarchical_mapper
  image_deleter
  image_filterer
  image_rectifier
  image_registrator
  image_undistorter
  image_undistorter_standalone
  mapper
  matches_importer
  model_aligner
  model_analyzer
  model_comparer
  model_converter
  model_cropper
  model_merger
  model_orientation_aligner
  model_splitter
  model_transformer
  patch_match_stereo
  point_filtering
  point_triangulator
  poisson_mesher
  project_generator
  rig_bundle_adjuster
  sequential_matcher
  spatial_matcher
  stereo_fusion
  transitive_matcher
  vocab_tree_builder
  vocab_tree_matcher
  vocab_tree_retriever

My modified demo config:

  # @package _global_
type: eval_dataset

# Ray configs:
ray:
  enable: False
  slurm: False
  n_workers: 4
  n_cpus_per_worker: 6
  n_gpus_per_worker: 1
  local_mode: False

n_scene: null # For debug
scene_list:
  - example_scene
sub_use_ray: False
sub_ray_n_worker: 1

verbose: False
visualize: False

dataset_base_dir: /home/dev/A/Pipelines/data/ # NOTE: please modify here!
dataset_name: testtrack
output_base_dir: SfM_metric_output
exp_name: ''

use_prior_intrin: False
method: DetectorFreeSfM

neuralsfm:
  # Method related:
  NEUSFM_enable_post_optimization: True
  triangulation_mode: False
  NEUSFM_coarse_matcher: 'loftr_official'
  NEUSFM_coarse_match_type: 'coarse_only'
  NEUSFM_refinement_chunk_size: 2000
  NEUSFM_coarse_match_round: null
  NEUSFM_coarse_match_thr: 0.2
  NEUSFM_fine_match_type: 'multiview'
  suffix: ''
  NEUSFM_fine_match_model_path:
    - weight/multiview_matcher.ckpt
  NEUSFM_fine_match_cfg_path: 
    - hydra_training_configs/experiment/multiview_refinement_matching.yaml

  # Data related:
  n_images: null # For debug
  img_resize: 1200
  img_preload: False
  down_sample_ratio: null
  img_pair_strategy: sequential
  INDEX_num_of_pair: null
  img_retrival_method: null

  # Controller:
  close_eval: True
  redo_all: False
  redo_sfm: False
  redo_matching: False
  redo_refine: False

  # Pose optimization related:
  refine_iter_n_times: 2

colmap_cfg:
  ImageReader_camera_mode: 'auto'
  ImageReader_single_camera: False
  min_model_size: 3
  no_refine_intrinsics: False
  use_pba: False
  n_threads: 16
  reregistration:
    abs_pose_max_error: 12
    abs_pose_min_num_inliers: 30
    abs_pose_min_inlier_ratio: 0.25
    filter_max_reproj_error: 5

  colmap_mapper_cfgs: null

Your assistance is greatly appreciated

hxy-123 commented 3 months ago

Hi, have you successfully compiled our modified version of COLMAP? If so, incremental_model_refiner can be seen by running colmap -h in the command line. Notably, if you only compiled by did not install the modified COLMAP (without running sudo make install), you are supposed to specify the COLMAP path by export COLMAP_PATH=/your/path/colmap/build/src/exe/colmap, as instructed in Install.md.