search

nianticlabs / mickey

[CVPR 2024 - Oral] Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences
https://nianticlabs.github.io/mickey/
Other
498 stars 31 forks source link

How to get overlaps.npz in customer datasets #7

Closed jywu511 closed 6 months ago

jywu511 commented 6 months ago

Hi, thanks for your great work. When I used my dataset, how can I get the overlaps.npz before training? I found it is important. Thanks!

Best regards

axelBarroso commented 6 months ago

Hello! The overlaps.npz was computed originally in the Map-free dataset. To obtain the overlap, they had to build the 3D models (using COLAMP).

Given two images (im0 and im1) and the 3D point cloud, you can check how many of the 3D points can be projected in both images. The overlap is the ratio between shared_3d_points and max(projected_3d_points_0, projected_3d_points_1). You can see more details in the Map-free supplementary material (look for Overlap Score section).

I found it is important.

Could you give more details on why overlaps are important? Are you using the curriculum learning configuration? Hopefully, I can assist a bit more.

jywu511 commented 6 months ago

Thanks for your response! I have understood the way getting overlaps.npz. However, if I want to train the mickey model with my own dataset, how should I generate my own overlaps.npz? Thanks.

axelBarroso commented 6 months ago

Hello!

Overlaps.npz are obtained with COLMAP (see my previous comment).

Regarding your dataset, do you have a list of images you want to use for training? And if so, do you know if they are solvable (or have any overlap)? If that is the case, I think that using the curriculum learning approach of MicKey should be enough for training. For instance, if you think that only 70% of the pairs are solvable, you could set 70 in here. That will let MicKey know that only the top 70% of the image pairs should be solvable, and that it can discard all the others.

jywu511 commented 6 months ago

Thanks for your response!