Sparse SfM tests

[albertopretto]

For each dataset:

• Perform sparse SfM by using COLMAP with single camera model with and without custom calibration (i.e., let COLMAP estimate the camera parameters in the first case, provide the camera parameters to COLMAP in the seconda case)
• For both the previous reconstruction, execute PP refinement
• Now we have 4 recosntrsction w/o calib parametrs and w/o refinement: try to align each reconstruction to google map, and qualitatively evaluate the reconstructions.

[gkiavash]

Here are the calibration parameters used for this task:

width: 5184
height: 3888
K: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 2.2880100739220425e+03, 0., 2.5716169994772490e+03, 0.,
       2.2845293722010042e+03, 1.9205706165878657e+03, 0., 0., 1. ]
D: !!opencv-matrix
   rows: 8
   cols: 1
   dt: d
   data: [ -2.6434196998339271e-01, 9.9571471417386093e-02,
       -2.4160314862664079e-04, -2.2267220647390027e-04,
       -1.9631169477800196e-02, 0., 0., 0. ]

The COLMAP bash code to consider these parameters can be found here: https://github.com/gkiavash/Master-Thesis-Structure-from-Motion/blob/cdcecff925e424ada3b6328ad2f331d81e51d933/scripts/colmap_sparse.sh#L16

• Note: calibration dataset has different dimensions than SfM dataset. Checkerboard images are captured with 5184*3888 pixels, while SfM images are taken from video frames with 2.7k quality. So, Cx and Cy parameters in the camera matrix should be changed. That was the reason COLMAP couldn't find the initial pair in our in-person session.
• Link to the calibration images: https://drive.google.com/drive/folders/199WSPGOkQ5wyquNwL-v2gRp1DSO1eXIs?usp=share_link

[gkiavash]

Final point clouds can be found here: https://drive.google.com/drive/folders/1L2aoLnA_Ug5gX-sfVuEyvkqMtplbcxNj?usp=share_link

File names	Before refine	After refine
without calibration params	o.ply	o_pp.ply
with calibration params	o_calib.ply	o_pp_calib.ply

• SfM pipeline with initial calibration parameters took 2 hours long to process. Finding an initial good pair and registering needed much more time than pipeline without initial calibration params
• Dataset details:
  • Test video: 58 seconds, 2.7k, 60 frames/sec, wide and distorted
  • Video to frames: every 20 frames, means 3 images per second
  • Sparse Reconstruction in all 4 cases: ~100k points

In order to compare with the city dataset, the path that the video is captured is shown on google map:

[gkiavash]

Different views of the point clouds: (Open the images to zoom in)

1. Above:

2. 

• City point cloud of the same view with different background color

3. 

• City point cloud of the same view with different background color

  • One useful piece of information from the top viewpoint is the angle of the streets and walls. However, after adding calibration parameters, the angles of the streets are corrupted in comparison to city ply and google map.
  • Comparing before and after pixel-perfect refinement, we have more points in open spaces, where have fewer walls. This is good since the city point cloud has almost no points for walls of buildings.

[gkiavash]

A closer look at "Piazza Cavour":

• City point cloud:

	Left	Right
Top	o.ply	o_pp.ply
Down	o_calib.ply	o_pp_calib.ply

[gkiavash]

I am going to create more point clouds from other parts of the city

[gkiavash]

Comparing COLMAP's auto-calibration and our manual calib params:

1. without prior calib params:
   • SIMPLE_RADIAL
   • (3244.8, 1352.0, 769.0, 0 )
   • (f, cx, cy, k)
2. with calib params:
   • FULL_OPENCV
   • (2288.01, 2284.52, 1352.0, 769.0, -0.264342, 0.099571, -0.000242, -0.000223, -0.019631, 0, 0, 0)
   • (fx, fy, cx, cy, k1, k2, p1, p2, k3, k4, k5, k6)
   • COLMAP doesn't refine the initial calib params since these params have not changed from the input.
   • link to calib dataset: https://drive.google.com/drive/folders/199WSPGOkQ5wyquNwL-v2gRp1DSO1eXIs?usp=share_link

• These data are taken from the colmap project's database:
• Pixel perfect hasn't refined calib params
• In both cases, I believe calib params are not good enough. The images are still distorted even after applying undistortion function
• I think the difference in camera settings(FOV, zoom, 2.7k vs 4k, etc. ) has weakened undistortion. Here is a comparison of the undistortion of calib dataset and urban dataset: (https://github.com/gkiavash/Master-Thesis-Structure-from-Motion/issues/7#issuecomment-1359428170, https://github.com/gkiavash/Master-Thesis-Structure-from-Motion/issues/7#issuecomment-1375329913). The checkerboard image has perfect straight lines, while the walls in the urban image still are curved, especially at corners.

[gkiavash]