How to use MultiCameraPose for 360 degree spherically captured scene with 3 cameras?

[kondela]

I came across this project after fiddling with COLMAP during last two months and still being unable to consistently get desired results on my specific use-case which relies on relative poses between cameras.

I am using COLMAP to estimate camera poses for my datasets which consist of hundreds of scenes that share some similarities. Specifically, each scene consists of spherically captured images (using a turntable) with a white background - in total either 144 images or 108 images per scene (3 or 4 vertical angles and 36 images capture horizontally per vertical angle).

Sample of images from one of the scenes:

Therefore, my camera rig consists of either 3 or 4 cameras (positioned roughly above each other) that share same intrinsics. For better illustration see screenshot from COLMAP's gui from a different scene with 3 cameras.

My current workflow is:

1. Extract features from images (colmap feature_extractor)
2. Match features across images (colmap exhaustive_matcher)
3. Take camera extrinsics from a previous scene where COLMAP was able to correctly estimate them
4. Run point triangulation using these extrinsics on the images
5. Run model refining (https://github.com/colmap/colmap/issues/1057#issuecomment-730266379 thank you once again @tsattler!)

However, sometimes this workflow fails at step 4 and 5 when the extrinsics taken from another scene are far too different - instead of a single coherent object it creates three different reconstructions of the same object, just misaligned in height.

Extrinsics may different in:

• Height from the ground of each individual camera might be adjusted (so although their relative height difference does not change a lot it is not a constant change across scenes)

Relative extrinsics that do not change:

• Camera in each vertical angle always takes 36 photos per round, i.e difference between shots in is always 10 degrees.
• Relative "height" difference between each camera within the scene is constant, i.e. cameras are perfectly above each other and their relative pose in height does not change within the scene.

I was wondering whether there's possibility to leverage these known relative poses using proposed Multi Camera Pose Estimation and if there is then how could one go about achieving it.

I managed to get the code running and currently I am in middle of converting COLMAP's cameras.txt and images.txt to images_with_intrinsics format based on common.h file. it would be awesome if there was a sample images_with_intrinsics file.

Also I am not sure if I understand common.h correctly but how do we include points2D and points3D from COLMAP?

Any help is appreciated :)

[tsattler]

The MultiCameraPose software assumes that a set of 2D-3D matches between pixel positions and 3D points in the scene is known for the cameras in a multi-camera rig. These matches are then used to estimate the pose (and potentially the scale of the rig). It can be used to estimate poses for new cameras with respect to a known 3D model. It cannot be used to reconstruct the scene itself.

If you are looking to reconstruct a scene given relative poses between images in a multi-camera rig, you could have a look at https://github.com/openMVG/openMVG . If I remember correctly, OpenMVG supports reconstructions with multi-camera rigs.

[kondela]

Thank you for the explanation! In the meantime COLMAP added some support for refining and estimating relative poses in global bundle adjustment and that partially solved my problem.