Localize Point Cloud with undistorted images

[gkiavash]

In this task, the camera is calibrated and the video is undistorted. I took several videos from the city and applied SfM to generate dense reconstructions. For some challenging reconstructions, I provided raw and refined dense plys to compare the results. The reconstruction from undistorted images had a much better quality than that from distorted images, like more accurate angles between streets, better aligned points on the ground and wall planes. Here are some of the point clouds and their corresponding points in the city point cloud:

1.

2.

3.

4.

[gkiavash]

Preprocessing

1. The city point cloud is perfectly aligned with the z-axis, however, our reconstructions are not. This is needed in order to create the elevation maps or slice the point clouds. So, for this problem, I detect the plane with the highest number of inliers to be the ground plane, i.e. xy plane and then calculated the rotation matrix between the normal of the ground plane and (0,0,1) pointer. The algorithm is robust enough since in most of the plys, all points of the ground are in one plane however buildings are in different planes. I also found a repository that segments the point cloud into multiple planes. In our case, it segments the ground and all buildings in different streets. The image below is an example of this segmentation.

2. Then, I created the elevation maps. Here are some the examples:

3. Another problem is the difference in scale and density of the points between the reconstructions and city plys. In city ply, the points are distributed uniformly. I scaled them manually and then filtered the points using open3d.geometry.voxel_down_sample function to have coherent and the same density between two plays

4. The dense points from the ground are important as the majority of the points in city point cloud are related to the ground and roofs. I recognized that Pixel Perfect refinement can generate more points for this area. Here is an example:

   • without refinement:
   • with refinement:

[gkiavash]

Registration

I tried to register using the methods in open3d.

• Registration using feature matching: after playing around with the algorithm and changing lots of parameters, I could register some of them. However, it is not a stable method:

• Registration using correspondence: I tried to detect the points related to crossroads in both plys as keypoints, and then register using all combinations of correspondence. This method works better. The rmse and fitness parameters in the evaluation can give us the best correspondence. However, the challenge is to detect the crossroads or any other point of interest shared in both plys, which I think is easier than the previous approach: