Point cloud too sparse

[karanpandyaa]

Hi, First of all thank you for this amazing repo!

I was able to run the whole thing using the sample data but when i am visualizing the point cloud in Open3D i am not able to see the structures properly like shown on the project website. What should i do to see it properly?

Also how do i get the camera intrinsics and extrinsic, i am only seeing the ground plane equation?

And how to see the scene from the camera view, i can just move around the point cloud, but not able to select the camera.

Thank you.

[kvuong2711]

Hello,

Thank you for your interest in our work!

I was able to run the whole thing using the sample data but when i am visualizing the point cloud in Open3D i am not able to see the structures properly like shown on the project website. What should i do to see it properly?

This point cloud is indeed a sparse point cloud. I think if you change the point size of Open3d (see some instructions here), you should be able to see the structures more clearly. To get the semi-dense reconstruction as shown in the website, we additionally run COLMAP's MVS (you can follow the instructions here). This is just for visualization purposes so I don't include it in the repo (really it's just following COLMAP's instructions linked above).

Also how do i get the camera intrinsics and extrinsic, i am only seeing the ground plane equation?

And how to see the scene from the camera view, i can just move around the point cloud, but not able to select the camera.

If you look at the groundplane_fitting code, you will see how to access the intrinsics/extrinsics of the cameras. If you prefer a GUI, perhaps a more straightforward way is to load the reconstruction using COLMAP, then you can visualize/select cameras/etc. easily with COLMAP (e.g., just need to open this folder using COLMAP).

Hope this helps!

[kvuong2711]

Closed due to inactivity. Please re-open if you have any other question, thanks!

[karanpandyaa]

Thank you, I was able to get the intrinsics and the dense reconstruction!

[karanpandyaa]

Just one small doubt, after the dense reconstruction i got a .ply point cloud which looks fine. But in your project video you have shown camera localized in that scene whereas in my point cloud file there is no camera, what should i do ?

Also, how did you localize the car in the scene, with the camera and the point cloud? I know it might be out of context of the paper but if you could point me to the repo or method how did you do it would be great! Thanks again!

This the the point cloud the dense reconstruction generated but no cameras in it.

[kvuong2711]

@karanpandyaa You can try loading the .ply file using Open3D, something like this, then also load the cameras and visualize it like in the script:

vis = o3d.visualization.Visualizer()
vis.create_window()
dense_scene_mesh_path = latest_sparse_model_path / 'fused.ply'
if dense_scene_mesh_path.exists():
    dense_scene_mesh = o3d.io.read_point_cloud(os.path.join(latest_sparse_model_path, "fused.ply"))
    [pcd_dense, _] = dense_scene_mesh.remove_statistical_outlier(nb_neighbors=50,
                                              std_ratio=1.0)
    vis.add_geometry(pcd_dense)  # pcd

For reconstructing cars, it's a mix-match of a few things, but there are a few different papers you can follow: 1) Detect 2D keypoints, then solve for the 6DoF pose with PnP: https://www.cs.cmu.edu/~ILIM/projects/IM/TRAFFIC4D/ 2) Newer methods, with feed-forward networks: https://github.com/lkeab/gsnet or https://arxiv.org/pdf/2209.05624

Hope this helps!

[karanpandyaa]

Thanks for the quick responses, I will try these out!