Registration of overlapping point clouds

[Reno989]

Hello everyone, I am currently working on my master thesis and would like to use this library for it. To be able to tackle my actual task, I first need to merge two point clouds. These two point clouds represent the ground in front of a vehicle and are obtained with two 3D cameras (see picture).

I know the rough orientation of the two cameras, so that I can roughly pre-transform the point clouds. As can be seen, the point clouds overlap in a certain area. The goal now is to create an overall cloud from the two individual point clouds.

The point clouds are very large (23232 points per cloud), so I use only the right or left partial image for the registration. This halves the amount of data. Afterwards I always summarize 9 points by averaging. Thus I receive two times 1276 points. Each point has an X, Y and Z coordinate.

This amount of data can be processed by my RaspberryPi.

If I now perform a registration (affine or rigid), the result deteriorates very strongly.

The code used looks like this:

# self.__teilbilderReduziert[i] contains the reduced point clouds. As the clouds are sorted by pixels, the shape will be adjusted
self.__links = np.reshape(self.__teilbilderReduziert[1], (1276,3))
self.__rechts = np.reshape(self.__teilbilderReduziert[0], (1276,3))  

self.__reg = RigidRegistration(**{'X': self.__links, 'Y': self.__rechts})
self.__reg.register()  

# self.__bildVorausgereichtet[i] includes the overall images. These are also sorted by pixels and are adjusted in shape 
self.__bildVorausgereichtet[1] = np.reshape(self.__bildVorausgereichtet[1], (23232,3))
self.__bildVorausgereichtet[1] = self.__reg.transform_point_cloud(self.__bildVorausgereichtet[1])

# Finally, the untransformed and the transformed image are saved
self.__bildVorausgereichtet[0] = np.reshape(self.__bildVorausgereichtet[0], (23232,3))
np.savetxt("Bild_rechts.txt", self.__bildVorausgereichtet[1])
np.savetxt("Bild_links.txt", self.__bildVorausgereichtet[0])

What can I do to solve my problem?

I would be very pleased about answers and help. With best regards

René

[siavashk]

Looking at your images, it is hard for me to visualize what the rigid solution for this registration problem would look like. As shown in the following two figures, it seems that the red and green regions of each point cloud correspond to each other. If that is correct, there is some form of reflection happening in your dataset. If this is the case, you need to remove the reflection in your point clouds prior to registration so that optimization becomes easier.

[Reno989]

Hello @siavashk , first of all thank you very much for your answer. However, I think there is a little misunderstanding there. Therefore I try to explain my data again a little better.

As already described, I am using two 3D cameras, each providing one "image". If I display the two images unchanged, the following picture results: (My sample images do not always show the same scenarios for technical reasons. But of course the respective individual images always fit together)

The camera test setup looks like this:

Since I roughly know the positioning and orientation of the two cameras to each other, I pre-transformed both images in the next step. Thus, the two images are approximately in the correct alignment to each other. This can be seen in the next pictures:

For example, the marked tips represent one and the same tip. Due to the pre-transformation, they are also in close proximity to each other, but not yet "exactly" on top of each other.

Since an image contains 23232 points, I cannot pass the entire point clouds to the library for processing. The working memory is clearly too small for this. For this reason, I perform a data reduction beforehand. On the one hand, I know that only the right half of the left image and the left half of the right image overlap.

So for registration I use only the two partial images. Since this reduction is still not sufficient, I reduce the points in the next step. By averaging I always combine nine blue points to one red point. Of course only for the corresponding partial image.

Thus, I finally get two point clouds / images corresponding to the right half of the left image and the left half of the right image, each with a reduced number of points. I pass these two point clouds to the registration method: self.__reg = RigidRegistration(**{'X': self.__links, 'Y': self.__rechts})

Following this, I apply the transform_point_cloud method to the entire right image. As you can see in the following image, the image that was already very close to the final alignment is clearly tilted and consumed.

I am now looking for a solution to this problem. I hope the data is now a little clearer and easier to understand.

I would be very happy about another answer. With best regards René

[gattia]

@siavashk, I was following along to see if I have thoughts but will defer to you here.

Im wondering if this has to do with the shrinking that occurs during the first step of the rigid registration process? I know this was reported here https://github.com/siavashk/pycpd/issues/15 for using the exact same point cloud twice, but I also find that I get this same behaviour whenever I use rigid registration. You can also see this shrinking occurring in the examples I made on the cycpd github: https://github.com/gattia/cycpd. On there, both of the affine and rigid transforms essentially get compressed to be just one point and then expand in size as time goes on. I've done some print statements to look at the data and it is definitely the scale factor that is getting squashed. This squashing to zero size works for closed manifolds. But I wonder if this is what is causing the weird results for planes of data like is shown here.

I know you mentioned in that other issue https://github.com/siavashk/pycpd/issues/15 that it was likely due to instabilities in the SVD decomposition. Considering my examples arent the same knee, it isn't fully that. Is there another potential thought you have? If this is a potential reason, I'm happy to look deeper into solutions and try some things out.

[gattia]

@Reno989 - Im curious if you think that forcing the scale to be constant in the registration might help your problem? Right now, presumably the problem is looking for the rotation/scale to best match the data. If you remove the scale part it might constrain the problem and help with the solution.

see : https://github.com/siavashk/pycpd/issues/34#issuecomment-1175339845

[gattia]

@Reno989 the option to turn off scaling has now been added (https://github.com/siavashk/pycpd/pull/67). Let us know if that helps.

Otherwise I'm closing this issue.

Alternatively - the new addition by @agporto here: https://github.com/siavashk/pycpd/pull/57 might be helpful as it allows the user to specify priors to help guide the registration.

[Reno989]

Hello @gattia , thank you very much for your support. Unfortunately, my project has been on hold for quite some time and I have not been able to continue working on it.

However, as soon as I am able, I will continue to pursue the approaches described here.

For now, though, it's perfectly fine to finish the process.

Thanks again to all the supporters.

With kind regards Reno989