Pose priors

[reynoldscem]

Hi,

If I have prior pose estimates can I incorporate them in any way? And is there any support for multi-camera setup? I.e. pairs of cameras with fixed relative poses

[ahojnnes]

(a) Using a multi-camera rig with fixed relative poses between the cameras in the rig is currently not a supported feature. (b) The usage of pose priors is currently not a well-tested and supported feature. The plan is to support and test this better in the future.

[reynoldscem]

Thank you! And thanks for the great work

[reynoldscem]

Shall I close, or leave open as an enhancement?

[pwais]

@ahojnnes

For (A) pose priors with tracks, there's been some recommendation to just accept the poses as registered and then run BA + triangulation ( https://github.com/colmap/colmap/issues/1057#issuecomment-730266379 ).

Separately for (B) unregistered images with pose estimates, one can include those poses in the DB though in my experience the incremental mapper typically ignores them (?) and it can be better to simply align the reconstruction with the input poses (e.g. fit location xyzs).

For (B) it seems that user pose priors could be at odds with GLOMAP's rotation averaging, and so would there actually be much benefit from accepting them? Perhaps you could give some color on how the COLMAP incremental mapper uses them (if at all ..).. Does this sound right?

For (A), perhaps the approach is to just run parts 6 (BA) and onward in the global mapper? ( https://github.com/colmap/glomap/blob/da77885ed90a8b5ab846ff10bb788662b7d7c244/glomap/controllers/global_mapper.cc#L162 ). And then I guess the user could benefit from the decoupled position / rotation BA rounds, depending on how the user poses compare to what GLOMAP would initialize. Does this sound right?

At the end of the day, it seems perhaps there might not be good datasets to validate the pose priors feature(s) ?

[sklappal]

Thanks for glomap - this is a really valuable contribution in this scene! I'm just commenting here that properly supporting pose priors would be reeaaally nice. I would hope that they could be used to speed up the process and also tackle outliers the mapping phase sometimes produces. Also relative transform priors could be useful, if that would be better from the implementation perspective :) The thing is that we often have some IMU data or online pose estimates available, and would want to use those as a starting point. I have not been able to make the suggested BA + triangulation work reliably.

[oliver-batchelor]

Aside from the potential to more robustly align scenes, pose priors are very useful for aligning a reconstruction with a real-world coordinate system (e.g., local GPS coordinates or coordinates from a robotic arm).

As far as I can tell, COLMAP currently does nothing with pose priors in the mapper, but there's a model aligner (I've never used it) to align to pose priors afterwards. I think the main thing it does is help speed up the selection of potentially matching images.

[pwais]

@sklappal Just curious but when you say "I have not been able to make the suggested BA + triangulation work reliably" do you mean the "unofficial" formula in https://github.com/colmap/colmap/issues/1057#issuecomment-730266379 ? Or something else?

@oliver-batchelor The poses will still drift a bit during BA, and so one may need to always run the model-aligner afterwards to make the reconstruction match the reference frame as closely as possible.

[oliver-batchelor]

@pwais One way to incorporate pose priors is in the optimization itself. I am fairly sure Metashape operates this way, the user provides reference poses + variances on the position and rotation. Otherwise you are right, they will drift and or change scale from the original reference positions.

[sklappal]

@sklappal Just curious but when you say "I have not been able to make the suggested BA + triangulation work reliably" do you mean the "unofficial" formula in colmap/colmap#1057 (comment) ? Or something else?

I have seen this piece of code but can't now remember how far I got with that. I think I managed to make it run, but it did not immediately produce the results I was hoping for - I did not do extensive testing with that.

I have also just run colmap point_triangulator and colmap bundle_adjuster successively-iteratively for multiple sequences, but it has turned out to be slower and less accurate than just doing incremental mapping.

I've also experimented with hierarchical mapper, but there the accuracy seems to suffer. I guess one can run global BA after that, but that tends to get slow as well..

My prior poses are from AR Kit (iPhone), and they do have severe jump corrections when there are loop closures. Running incremental mapper typically works quite well, except that it gets very slow. It can also produce severe outliers if there are repeated patterns. I would hope that if I could constrain the solution with priors, all my problems would disappear 😇

[lpanaf]

Hi @sklappal, we have tried incorporating pose prior in this branch: https://github.com/colmap/glomap/tree/user/joschonb/pose-prior-fix. To use this, you would need to put the pose prior in the database file, set the --constraint_type to be ONLY_POINTS (which is the default), and set --GlobalPositioning.optimize_positions to be 0. It should skip the camera center optimization in the global positioning part by directly using the ones provided by the pose prior. If desired, the further global bundle adjustment can also be skipped (this will sacrifice the accuracy though). Note that this feature has not been well-tested, so we would appreciate it a lot if you could give us some feedback on it.

[sklappal]

Hi! Yes I can test it. Does this support only the translational part, or the rotation as well? I have been using COLMAP 3.8 but it seems the db schema has changed a bit. There used to be priors for both rotation and translation in the images table, but now there is only the translational part in a pose_priors table.

[lpanaf]

GLOMAP only needs the camera location. It will estimate rotation in the process

[ahojnnes]

@sklappal Any reason for using colmap 3.8 vs a newer version?

[sklappal]

Thanks. No particular reason, it's just the version that we were using. I'm now updating it to the commit glomap is using.

[sklappal]

What would be the expected or intended effect of using these positional priors? More robust results? Less outliers? Faster runtime? Something else?

[sklappal]

I was able to run the code in the branch, but I'm not getting the results I'm expecting.

The commandline I'm using is glomap mapper --constraint_type ONLY_POINTS --GlobalPositioning.optimize_positions 0 --database_path <db> --image_path <images> --output_path <output>

The pose priors I'm setting are AR Kit positions transformed to the COLMAP convention. I'm filling the pose_priors as follows:

cursor.execute("INSERT INTO pose_priors VALUES (?, ?, ?)", (colmap_id, array_to_blob(colmap_position), coordinate_system))

the colmap_position is in the camera coordinate space (as opposed to world space) as that seems to be the COLMAP convention. But this should not really matter. The coordinate space value I write to the database is -1.

Below is a graph that shows the relative transforms (or their magnitudes) between the priors and the glomap mapper output:

There are quite a few outliers. Without the pose priors I get stable results. These results happened on both of the two datasets I tested with.

[lpanaf]

Thanks for the experiments. Can you maybe share more on what results you are seeing? Is everything completely wrong or it is only having a few outliers? If it is the latter case, I am not sure whether it is a problem with the code or with the prior. To debug this, you can also skip the bundle adjustment and retriangulation step by also setting --skip_bundle_adjustment arg 1 --skip_retriangulation 1 and continue using --GlobalPositioning.optimize_positions=0 flag. It should directly use the camera centers provided with pose prior and only optimize 3D points, so the camera center should be the prior

[sklappal]

Thank you for the help. Could you describe a bit what is the intended use of the pose priors? Is it using the positions as starting point and then optimizing those, or is it keeping the positions fixed and just optimizing the rotations and 3D points?

The pose priors are from AR Kit SLAM which are not fully accurate, so I would not want to to keep those fixed. I'm hoping that pose priors could be used for speeding up the process and rejecting some obviously incorrect outliers.

[lpanaf]

Hi @sklappal, regarding the intended use, it depends on the nature of your data. As you said, if you think the prior is not accurate enough, then you can use it as an initialization and optimize from there. To achieve this, currently, there is no direct way to run it from the command line. You will need to change the generate_random_positions to false in the global positioning option. Then the pose prior will be used as a starting point, instead of having random positions. However, I am unsure how much time-wise benefit it can bring, as the optimization can already converge in a few iterations. What I suggested before was to directly skip the position optimization in the global positioning step, and only optimize them in the bundle adjustment. This should save you quite some time, and I would guess the performance gap is not very large. So I would recommend you to try the method I suggested before, and if the result is unsatisfying, then you can go with the first method :)

[h1063135843]

GLOMAP only needs the camera location. It will estimate rotation in the process

The rotation is esitimated with the first camera fixed according to RotationEstimator::SetupLinearSystem function. The prior camera location should also be in the first camera coordinate system.

With given position prior, the sim3 which transform sfm result to world is determined. The camera location can't be fixed without changing rotation estimated if their coordinate axis are not consistent.

[lpanaf]

Hi @h1063135843, thanks for the comment. You made a fair point here. In this case, we would need to estimate the extra aligning matrix.