Colored point cloud registration fail to achieve local best solution and a possible solution.

[lvgeng]

Colored point cloud registration is a really amazing feature of open3d... http://www.open3d.org/docs/release/tutorial/Advanced/colored_pointcloud_registration.html

There is a similar problem. As mentioned, the point clouds fly away. I checked the objective function and I think it is certainly a bug in this case.....

A wild guess is that the optimiser used in the registration have a larger step than expected. When the first guess is made, there is a possibility that the two point cloud has zero corresponding point pairs so that the objective function gives zero, which is considered to be a "better" solution than others. It might happen to a lot of situation, especially when the point clouds are relatively small and have few points.

https://github.com/intel-isl/Open3D/issues/362

However I have some issue while playing with my own data. The init transformation should be good enough I guess... The color difference is added in the visualization stage, it should not affect the colored point cloud registration.

My testing code and data python_script_and_data.zip

and results. (each one use the previous result as the initial pose) original: point to point point to plane registration_colored_icp

I understand that it is possible to have a bad result if the data is not so good, but I cannot understand why it pushes the point cloud away after the registration. Is there any solution for that?

My data is here. They are point cloud I generated by other methods.

And the function

result_icp = o3d.registration.registration_colored_icp( source_down, target_down, radius, current_transformation, o3d.registration.ICPConvergenceCriteria(relative_fitness=1e-6, relative_rmse=1e-6, max_iteration=iter)) should I change the relative_fitness or relative_rmse? And ... what do they do exactly?

[syncle]

Duplicate of #1261. Please check out this

[lvgeng]

Checked. Unfortunately I have already tried that... the details are in that issue.

Duplicate of #1261. Please check out this

[Fred3D-tech]

yes.. without a working ransac or other icp registration, colored point cloud registration makes pointclouds flying away...

[lvgeng]

yes.. without a working ransac or other icp registration, colored point cloud registration makes pointclouds flying away...

However... the problem is, when we are using colored point cloud registration, it means the ICP does not work well.

[germanros1987]

@lvgeng I agree. We definitely need to work on a new registration method/an upgrade of ICP. We will keep you posted

[szx0112]

Colored ICP is a necessary step for many reconstruction applications where the geometric difference between RGBD frames are not significant, along walls, inside a tunnel. Under such cases, the color information is the major source to guide the registration.

I think the flying away issue arise when we keep reducing the scale for finer registration. My workaround is to break the registration at certain scale when the fitness/inlier_rmse is smaller/larger than the previous iteration. This can prevent the registration diverge but since we break the registration at larger scale,we may not able to get the expected level of accuracy.

I really appreciate if someone can work on this and further improve the current colored ICP implementation in open3d.

[theNded]

According to my experience, Colored ICP will face numerical issues when the structure is very simple, like this or planar cases. We need to investigate that part and eliminate unreliable color gradients.