search

naver / kapture

kapture is a file format as well as a set of tools for manipulating datasets, and in particular Visual Localization and Structure from Motion data.
BSD 3-Clause "New" or "Revised" License
466 stars 66 forks source link

Gangnam dataset #35

Closed Grem-Lin closed 2 years ago

Grem-Lin commented 2 years ago

Hi I feel difficult to run kapture-localization pipeline on Gangnam dataset and get the same number on benchmark. I summarized my issue below, it would be great if you can take a look at this and help me!! Thanks.

  1. I downloaded the data using kapture_download_dataset.py following this instruction. I can do mapping using kapture-loc pipeline (after fixing some dataset path issues). BUT I am unable to do localization for query images because some dataset internal folders path are inconsistent with your code. For example, your code believe that query image and map image are under same folder, with only one trajectories.txt, records_camera.txt, but the downloaded Gangnam dataset has separate trejectories.txt, record_camera.txt for training/test/validation folder. I feel just fixing these myself doesn't make sense as I didn't see any tutorial/documentations about how to convert the downloaded Gangnam dataset into a proper format such that we can run kapture-loc on it. Here are actually some other issues (e.g. input argument conflicts, assert k_data.trajectories is not None etc.) Could you please let me know how to run kapture-loc on Gangnam successfully?

  2. Given the first issue, I tried to reproduced kapture-loc on Gangnam dataset myself, but I just got [0, 0, 0] as final evaluation result. I used 3D model build from kapture-loc, and I used r2d2, APGeM, geometric verification and colmap registration for localization. Currently I didn't see any bugs in the code (though I believe that I must miss something): r2d2 matching looks reasonable, gv works, register works. Could you point me some potential issues? And could you let me know how to get that result as on benchmark?

Thank you so much for the time!! JS

humenbergerm commented 2 years ago

Hi!

There shouldn't be some dataset path issues. So if there are some, we need to find them. Would you mind having a look at these instructions (they use this script) and telling us at which steps you encounter problems?

Best, Martin

Grem-Lin commented 2 years ago

Hi, Thanks for the points! I didn't check the code recently and it turns out that the new document was released 24 days ago! I pulled the newest code and run it. I successfully re-downloaded the dataset, but I failed at merge folder step and the following Extract global features steps etc. are all fail accordingly. I copied my errors for you (note that I only run B2 dataset to debug, my bash script and full error report are here):

INFO ::merge: Loading /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping DEBUG ::kapture: loading sensors /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping/sensors/sensors.txt ... DEBUG ::kapture: loading rigs /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping/sensors/rigs.txt ... DEBUG ::kapture: loading trajectories /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping/sensors/trajectories.txt ... DEBUG ::kapture: 4 518 <class 'kapture.core.Trajectories.Trajectories'> in 0.048 seconds DEBUG ::kapture: loading images /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping/sensors/records_camera.txt ... DEBUG ::kapture: 4 518 <class 'kapture.core.Records.RecordsCamera'> in 0.007 seconds DEBUG ::kapture: loading depth /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping/sensors/records_depth.txt ... DEBUG ::kapture: 4 518 <class 'kapture.core.Records.RecordsDepth'> in 0.008 seconds DEBUG ::kapture: Loaded in 0.069 seconds from "/home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/mapping" INFO ::merge: Loading /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/test DEBUG ::kapture: loading sensors /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/test/sensors/sensors.txt ... DEBUG ::kapture: loading images /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/test/sensors/records_camera.txt ... DEBUG ::kapture: 1 786 <class 'kapture.core.Records.RecordsCamera'> in 0.003 seconds DEBUG ::kapture: Loaded in 0.005 seconds from "/home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/test" DEBUG ::kapture: Linking data files 4518it [00:00, 19689.19it/s] DEBUG ::kapture: Linking data files 1786it [00:00, 19430.06it/s] Traceback (most recent call last): File "/home/sadie/anaconda3/envs/kapture/bin/kapture_merge.py", line 146, in <module> merge_command_line() File "/home/sadie/anaconda3/envs/kapture/bin/kapture_merge.py", line 142, in merge_command_line merge_kaptures(args.inputs, args.output, args.keep_sensor_ids, args.image_transfer, args.skip, args.force) File "/home/sadie/anaconda3/envs/kapture/bin/kapture_merge.py", line 94, in merge_kaptures merged_kapture = merge_remap(kapture_data_list, skip_list, File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/kapture/algo/merge_remap.py", line 455, in merge_remap new_matches = merge_matches_collections(matches, data_paths, kapture_path, tarcollection_list) File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/kapture/algo/merge_reconstruction.py", line 291, in merge_matches_collections assert len(keypoints_types) > 0 AssertionError Launching on GPUs 0 global_features_type set to Resnet101-AP-GeM-LM18 /home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.decomposition.pca module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.decomposition. Anything that cannot be imported from sklearn.decomposition is now part of the private API. warnings.warn(message, FutureWarning) /home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/sklearn/base.py:313: UserWarning: Trying to unpickle estimator PCA from version 0.20.2 when using version 0.22.1. This might lead to breaking code or invalid results. Use at your own risk. warnings.warn( => loading checkpoint 'dirtorch/data/Resnet101-AP-GeM-LM18.pt' (current_iter 376) loading /home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/map_plus_test/ Traceback (most recent call last): File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/sadie/sadie/kapture-gangnam/deep-image-retrieval/dirtorch/extract_kapture.py", line 149, in <module> res = extract_kapture_global_features(args.kapture_root, net, args.global_features_type, File "/home/sadie/sadie/kapture-gangnam/deep-image-retrieval/dirtorch/extract_kapture.py", line 33, in extract_kapture_global_features kdata = kapture_from_dir(kapture_root_path, None, File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/kapture/io/csv.py", line 1472, in kapture_from_dir assert kapture_data.__version__ == current_format_version() AssertionError ./run_gangnam.sh: line 63: move: command not found mv: cannot stat '/home/sadie/sadie/kapture-gangnam/GangnamStation/B2/release/map_plus_test/reconstruction/global_features/Resnet101-AP-GeM-LM18/*': No such file or directory Btw, you may also want to comment out L63.

Could you please let me know how to fix this merge issue? Thank you so much!

Thanks,

Grem-Lin commented 2 years ago

I studied the kapture_merge.py. Shall I add some skip list for it? Or did I miss something? Thanks.

yocabon commented 2 years ago

Hi, from what I can see, you're encountering a bug that was fixed a few months ago: see this commit

File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/kapture/algo/merge_remap.py", line 455, in merge_remap
    new_matches = merge_matches_collections(matches, data_paths, kapture_path, tarcollection_list)
  File "/home/sadie/anaconda3/envs/kapture/lib/python3.9/site-packages/kapture/algo/merge_reconstruction.py", line 291, in merge_matches_collections
    assert len(keypoints_types) > 0
AssertionError

these lines numbers correspond to before the fix. I suggest you retry with both kapture and kapture-localization to their latest version.

Grem-Lin commented 2 years ago

Hi,

Thanks for the reply!! I updated everything to the newest. Now I have: kapture 1.1.5 and kapture-localization 0.1.4. I got kapture_merge.py, APGeM and r2d2 worked. I have two further questions here:

  1. I am unable to use your set parameters for r2d2 with max_size 9999 due to CUDA out of memory issue and I set it as 2048. Would it cause any issues for visual localization accuracy? Like it will have less-quality r2d2 feature and less accurate 3D model, and finally worse pose result?
  2. I am unable to get colmap point_triangulator work. Importing keypoint, matches, and triangulate all works, only BA step fails. I got some error like this: image and this (I just do colmap point_triangulator in terminal): image

It seems that the process was killed (I tried 3 times). Do you have any ideas of why? I checked the previous similar issue from kapture-localization and solution is out-of-date colmap version, but my colmap version is 3.7 (newest one). And I can actually triangulate the 3D model with exact same dataset B2 (the only difference is that my previous r2d2 parameters are: --min-scale 0 --min-size 256 --max-size 1024 --top-k 5000, while currently is --min-scale 0.3 --min-size 128 --max-size 2048 --top-k 20000), I believe there are more 3D points now, maybe it causes the fail due to some memory issues?

Thanks, JS

yocabon commented 2 years ago

Hi, about r2d2 max-size, yes it may negatively impact the localization accuracy. by how much I wouldn't be able to say, usually not that much.

for the bundle adjustment crash, the COLMAP process very likely ran out of ram. the bundle adjustment part of the mapping process is quite memory hungry. (same error in this issue on a different dataset).

Grem-Lin commented 2 years ago

Hi, I see your point. May I ask how much RAM and how much memory of GPU do you think are doable for this dataset? I really hope to get kapture-loc work on Gangnam dataset. But due to my machine limitation, I am not able to do it. I am wondering if you could share us the built model from colmap? I am assuming that image register will not take too much memory, please correct me if I am wrong.

Thanks, JS

humenbergerm commented 2 years ago

Hi!

We usually have about 512GB of RAM on our machines. If we use machines with lower RAM we also encounter crashes. We cannot share the colmap model because we would also need to share the descriptors and this is too much data. Since we are aware that this is a limitation of such methods, we hope to soon release a more memory efficient version of R2D2.

Best, Martin

yocabon commented 2 years ago

Hi, for the max_size issue you can try faster2d2_WASF_N8_big.pt, it should require less vram than r2d2_WASF_N8_big.pt and give comparable localization accuracy. It won't help with the ram issue of the bundle adjustment though.

Grem-Lin commented 2 years ago

Hi, Thanks for the reply! I will try faster2d2 then! Btw, I finally do 5000 keypoints per image and the results for B2 are: image I think it looks very reasonable now!

Thanks again for the help!

JS