Reproducing Results on the https://www.visuallocalization.net/details/17831/ Benchmark

[fulkast]

Hi @Skydes

Thank you for sharing your implementation and the tools surrounding this localization framework!

I have been trying to reproduce the results of hloc + pixloc on the visual localization benchmark for the Extended CMU dataset. However, I haven't been able to get results close to the values seen on the linked benchmark. The values I'm currently getting are:

Locally, I've downloaded the pixloc_cmu pre-trained weights hosted here, and I'm running the following command: python -m pixloc.run_CMU --from_poses Which after hours of running terminates with the message (truncated): [11/25/2021 00:32:21 pixloc INFO] Finished evaluating all slices, you can now submit the file /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice2-3-4-5-6-13-14-15-16-17-18-19-20-21.txt to https://www.visuallocalization.net/submission/ I'm assuming that --from_poses runs the evaluation using hloc poses as a start, is this correct? Also, do you have any pointers on what I must be doing wrong?

[patelajaychh]

$  python -m pixloc.run_Cambridge

Std Output - ( Only for KingsCollege scene )

[11/26/2021 18:46:06 pixloc INFO] Parsed configuration:
experiment: pixloc_megadepth
features: {}
optimizer:
  num_iters: 100
  pad: 2
refinement:
  num_dbs: 5
  multiscale:
  - 4
  - 1
  point_selection: all
  normalize_descriptors: true
  average_observations: true
  filter_covisibility: false
  do_pose_approximation: false

[11/26/2021 18:46:06 pixloc INFO] Working on scene KingsCollege.
[11/26/2021 18:46:06 pixloc.localization.model3d INFO] Reading COLMAP model /home/ajay/pixloc/outputs/hloc/Cambridge/KingsCollege/sfm_superpoint+superglue.
[11/26/2021 18:46:08 pixloc.utils.io INFO] Imported 343 images from query_list_with_intrinsics.txt
[11/26/2021 18:46:08 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/26/2021 18:46:11 pixloc.localization.localizer INFO] Starting the localization process...
 47%|██████████████████████████████████████████▉                                                | 162/343 [12:05<11:33,  3.83s/it][11/26/2021 18:58:18 pixloc.localization.base_refiner INFO] Optimization failed for query seq2/frame00033.png
100%|███████████████████████████████████████████████████████████████████████████████████████████| 343/343 [23:46<00:00,  4.16s/it]
[11/26/2021 19:09:57 pixloc.utils.io INFO] Writing the localization results to /home/ajay/pixloc/outputs/results/pixloc_Cambridge_KingsCollege.txt.
[11/26/2021 19:09:57 pixloc INFO] Evaluate scene KingsCollege: /home/ajay/pixloc/outputs/results/pixloc_Cambridge_KingsCollege.txt
[11/26/2021 19:09:57 pixloc.utils.eval INFO]
Median errors: 0.128m, 0.228deg
Percentage of test images localized within:
        1cm, 1deg : 0.29%
        2cm, 2deg : 1.46%
        3cm, 3deg : 4.37%
        5cm, 5deg : 14.29%
        25cm, 2deg : 73.47%
        50cm, 5deg : 87.76%
        500cm, 10deg : 96.21%

Here the med errors are 12.8cm/0.228deg which are different from what is claimed in paper ( 14cm/0.24deg, 13cm/0.24deg with oracle prior ). Please review this and help us to understand reason for this deviation.

Similarly, 7Scenes results are also different than claimed.

[sarlinpe]

@fulkast: This is indeed abnormal, let me run this again and dig into it. Did you also run the evaluation without --from_poses? if yes, what results do you obtain? With the recent code cleanup the recall should be around:	88.9 / 91.0 / 94.6	81.1 / 82.4 / 85.6	62.4 / 64.2 / 69.5

Which is slightly higher than what is reported in the original paper.

@patelajaychh Please don't hijack this issue, instead open a new one to discuss the results of Cambridge/7Scenes. The README explicitly mentions that the results might differ slightly, and generally improve rather than get worse. Given the rounding, the results that you obtain are rather consistent. Does it consistently get worse on other scenes? What is the difference observed with 7Scenes?

[fulkast]

Hi, @Skydes. Yes, I've also run the evaluation without the flag --from_poses and the results are on the third last row of the image from my previous post:

31.4/60.5/94.0	0.6/3.3/ 69.2	0.4/2.2/55.3

Does this slice list look correct for the evaluation on the Extended CMU dataset: pixloc_CMU_slice2-3-4-5-6-13-14-15-16-17-18-19-20-21.txt?

[sarlinpe]

On my end running python -m pixloc.run_CMU --slices 2-6 gives 89.0 / 91.1 / 94.6 for the urban category, consistent with my previous message. The slice list that you report is fine.

1. Can you dump the versions of all the main libraries? torch, numpy, opencv
2. Can you share the full console output, including the progress bars?

[fulkast]

Thanks for checking this! I will re-run the experiment and report back.

[fulkast]

Hi @Skydes,

Following your suggestion, I've re-run the evaluation on the urban category and got the results: 31.5 / 60.7 / 94.1 (pretty close to the results I previously reported). Below is the information that you asked for, please let me know if anything looks unexpected.

1) Pip freeze on the main libraries shows the following: numpy==1.17.4 opencv-python==4.5.4.58 torch==1.10.0+cu113 I will also attach the full pip freeze output here.

2) Here's the full output of my run on the urban category:

python3 -m pixloc.run_CMU --slices 2-6  experiment=pixloc_author_reference > evaluation_run.txt
[11/28/2021 23:02:54 pixloc INFO] Parsed configuration:
experiment: pixloc_author_reference
features: {}
optimizer:
  num_iters: 100
  pad: 2
refinement:
  num_dbs: 2
  point_selection: all
  normalize_descriptors: true
  average_observations: false
  filter_covisibility: false
  do_pose_approximation: false

[11/28/2021 23:02:54 pixloc INFO] Working on slice 2.
[11/28/2021 23:02:55 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice2/sfm_superpoint+superglue/model.
[11/28/2021 23:02:56 pixloc.utils.io INFO] Imported 3655 images from queries_with_intrinsics.txt
[11/28/2021 23:02:56 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/28/2021 23:03:01 pixloc.localization.localizer INFO] Starting the localization process...
100%|█████████████████████████████████████████████████████████████████| 3655/3655 [1:11:41<00:00,  1.18s/it]
[11/29/2021 00:14:42 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice2.txt.
[11/29/2021 00:14:43 pixloc INFO] Working on slice 3.
[11/29/2021 00:14:43 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice3/sfm_superpoint+superglue/model.
[11/29/2021 00:14:44 pixloc.utils.io INFO] Imported 5074 images from queries_with_intrinsics.txt
[11/29/2021 00:14:44 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 00:14:49 pixloc.localization.localizer INFO] Starting the localization process...
100%|█████████████████████████████████████████████████████████████████| 5074/5074 [1:42:21<00:00,  1.21s/it]
[11/29/2021 01:57:11 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice3.txt.
[11/29/2021 01:57:12 pixloc INFO] Working on slice 4.
[11/29/2021 01:57:12 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice4/sfm_superpoint+superglue/model.
[11/29/2021 01:57:13 pixloc.utils.io INFO] Imported 4728 images from queries_with_intrinsics.txt
[11/29/2021 01:57:13 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 01:57:18 pixloc.localization.localizer INFO] Starting the localization process...
100%|█████████████████████████████████████████████████████████████████| 4728/4728 [1:30:27<00:00,  1.15s/it]
[11/29/2021 03:27:45 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice4.txt.
[11/29/2021 03:27:45 pixloc INFO] Working on slice 5.
[11/29/2021 03:27:45 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice5/sfm_superpoint+superglue/model.
[11/29/2021 03:27:46 pixloc.utils.io INFO] Imported 2369 images from queries_with_intrinsics.txt
[11/29/2021 03:27:46 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 03:27:49 pixloc.localization.localizer INFO] Starting the localization process...
100%|███████████████████████████████████████████████████████████████████| 2369/2369 [44:19<00:00,  1.12s/it]
[11/29/2021 04:12:08 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice5.txt.
[11/29/2021 04:12:08 pixloc INFO] Working on slice 6.
[11/29/2021 04:12:08 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice6/sfm_superpoint+superglue/model.
[11/29/2021 04:12:10 pixloc.utils.io INFO] Imported 2547 images from queries_with_intrinsics.txt
[11/29/2021 04:12:10 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 04:12:15 pixloc.localization.localizer INFO] Starting the localization process...
100%|███████████████████████████████████████████████████████████████████| 2547/2547 [51:30<00:00,  1.21s/it]
[11/29/2021 05:03:45 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice6.txt.
[11/29/2021 05:03:45 pixloc INFO] Finished evaluating all slices, you can now submit the file /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice2-3-4-5-6.txt to https://www.visuallocalization.net/submission/

For more context, I'm running on commit d5f7f7d02910b9ac4c0289360c1, and here's a snapshot of my git log:

[sarlinpe]

What is the pixloc_author_reference experiment? Does it mean that you are evaluating weights that you retrained rather than the pretrained model? This was not clear from your description. If yes, can you share the training curves? do they look consistent with those of our reference experiment? Are you able to visualize the predictions on CMU in the demo notebook?

[fulkast]

Hi! Sorry, I should have elaborated on the last post. pixloc_author_reference is a folder I created to store the downloaded pre-trained weights, retrieved from here. So here's my outputs directory structure, where pixloc_cmu is my training experiment and pixloc_author_reference only has the one downloaded checkpoint_best.tar:

To double-check, here's the sha1sum of the checkpoint_best.tar file that I downloaded: sha1sum code: 0f33613a21bcb898dfdf469273d6a20ed7dc15f9

That said, I will share the loss curves from my training experiment (the one called pixloc_cmu), in case there is something anomalous there. And, for completeness, here's the tf-events file: events.out.tfevents.1637452030.the-machine.5052.zip

Please let me know in case anything looks suspicious.

[Edit]:

If possible, may I ask what metrics you get when you run python -m pixloc.run_CMU --slices 2. By isolating it to slice 2, I will be able to iterate faster in trying to find the error on my end.

Best, Frank

[ccy-ustb]

hello，@fulkast when i run “python3 -m pixloc.run_Aachen --from_poses” meet an error：

caochengyang@caochengyang-Lenovo-Legion-R9000P2021H:~/pixloc$ python3 -m pixloc.run_Aachen --from_poses True [12/02/2021 10:54:15 pixloc INFO] Parsed configuration: experiment: pixloc_megadepth features: preprocessing: resize: 1600 optimizer: num_iters: 50 pad: 1 refinement: num_dbs: 5 min_points_opt: 100 point_selection: inliers normalize_descriptors: true average_observations: true layer_indices:

0 1 [12/02/2021 10:54:15 pixloc.localization.model3d INFO] Reading COLMAP model /home/caochengyang/pixloc/outputs/hloc/Aachen/sfm_superpoint+superglue. [12/02/2021 10:54:27 pixloc.utils.io INFO] Imported 824 images from day_time_queries_with_intrinsics.txt [12/02/2021 10:54:27 pixloc.utils.io INFO] Imported 98 images from night_time_queries_with_intrinsics.txt [12/02/2021 10:54:27 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar [12/02/2021 10:54:32 pixloc.localization.localizer INFO] Reading hloc logs... Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/caochengyang/pixloc/pixloc/run_Aachen.py", line 77, in main() File "/home/caochengyang/pixloc/pixloc/run_Aachen.py", line 66, in main localizer = PoseLocalizer(paths, conf) File "/home/caochengyang/pixloc/pixloc/localization/localizer.py", line 145, in init with open(paths.log_path, 'rb') as f: AttributeError: 'Paths' object has no attribute 'log_path'

Have you encountered it? Can you help me solve it? grateful！！！

[fulkast]

@caochengyang0828 I will respond to the new issue you started.

[sarlinpe]

@fulkast This all looks good, my setup seems to match on every point, so I really have no idea of what's happening here. Here is what I get for slice 2 only: 17.6 / 17.9 / 19.2

[fulkast]

Thanks, @Skydes. Yeah, I get 7.3 / 13.1 / 19.2 for using only slice 2. Would it be okay to share the pixloc_CMU_slice2.txt file you used for the slice 2 submission? I want to take a closer look at my culprit cases. If you'd prefer to send them privately, this is my email: fulkast@gmail.com

[VictorZoo]

Got the same situation with you. And got the simiar results to yours. python -m pixloc.run_CMU

[sarlinpe]

• Results for Slice 2: pixloc_CMU_slice2.txt
• Output of pip freeze: pip_freeze.txt

Again, this is obtained by running the following on a fresh clone of the repo:

pip3 install -r requirements.txt
python -m pixloc.download --select CMU checkpoints --CMU_slices 2
python -m pixloc.run_CMU --slices 2 --results outputs/results/pixloc_CMU_slice2.txt

[fulkast]

Hi @Skydes, Thanks again for looking into this. While I don't yet have a sense of the cause of the problem, I can now at least reproduce your results on one of my machines. Both authors_pixloc_slice_2_6_home_device and authors_pixloc_slice_2_6_office_device were generated using the same command python3 -m pixloc.run_CMU --slices 2-6 experiment=pixloc_author_reference with the same setup (at least what I believe to be the same. Clearly they aren't), on my home and work computers respectively. Next, I will take a deeper look at the different between the two setups and report my findings.

[VictorZoo]

Looking forward to the final solution to this problem that has been bothering me for a long time. T.T Thank you very much @fulkast

[ccy-ustb]

hello,@fulkast,@Skydes. After i update the repository,on 7Scenes datasets,i got the similar results with paper. but on CMU datasets,i run :

pip3 install -r requirements.txt python3 -m pixloc.download --select CMU checkpoints --CMU_slices 2 python3 -m pixloc.run_CMU --slices 2 --results outputs/results/pixloc_CMU_slice2.txt

I still get this result：

Is this the reason for relying on the library version? This is my output of pip freeze: pip_freeze (1).txt

Thank you so much for answering this question, it has bothered me for a long time！

[fulkast]

Happy holidays all! I am away from work now and won't be making physical progress on this issue. In the meantime, however, I did come across this pytorch issue that looks like it might be related https://github.com/pytorch/pytorch/issues/70162. I will report more on this in the new year.

[fulkast]

Hi all! A happy new year to you all and I've got potentially some good news :)

In implementing some local tests I noticed that the results of my tensor matmul operations can be significantly different, depending on whether they were run on the GPU or the CPU. Long story short, this led me to this note here: https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices which was also relevant to this PyTorch issue: https://github.com/pytorch/pytorch/issues/67185

After setting

torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False

at the top of run_CMU.py my tests on a small subset of the data is looking more consistent with what @Skydes gets on slice 2 of the Extended CMU Seasons dataset. I'm now running on the full dataset and will report back soon.

[fulkast]

Hi @Skydes

I'm happy to share that I've been able to reproduce your results on slice 2 and I'm hence confident that I should be able to reproduce the results for the rest of the dataset as well.

It came down to the pytorch issue mentioned in this link. Essentially, I was already getting positional Euclidean errors in transforming 3-D points to the camera frame on the order of 10cm because I by default torch.backends.cuda.matmul.allow_tf32 = True on my RTX GPU.

Thank you for your feedback on my questions; your data was helpful in letting me hone in on the source of the problem!

Best, Frank