Partial optimal transport for shape assembly

[ttsesm]

Hi @Baio0,

I am interested to use Optimal Transport for shape assembly.

The idea in shape assembly is that you have a part of two different pieces which complete each other so that you can assembly a single merged piece. The issue is that you do not have any overlap part but only two fractured parts where there is a bijection between the two.

Thus, I am trying to understand whether SOPT could be used for such a use case or not.

Thanks.

[yikun-baio]

@ttsesm
Hi, Can I understand the problem in this way? For example: X={x_1,x_2,....x_100} is the source point cloud. Y={y_1,y_2,....y_200} is the target point cloud. Say you plan to assemble X_1={x_1,...x_10} (the set of red points) and Y_1={y_1,...y_10} (the set of blue points) which are subsets of X, Y, respectively. Your model (deformation function) is f(x)=Rx+\beta (I do not see source and target have different scaling), where R is rotation, and beta is translation (shift).

I think you can modify the code (remove the scaling s) and try. The sliced-opt method can work if you know how much mass you want to assemble (in this example, it is 10).

The challenges are:

1. The angle between the source and target point clouds is large (about \pi/2 in your example). To achieve a better performance, it is better if you have prior knowledge and apply it to initilize rotation R.
   (If you do not have prior knowledge, based on my knowledge, all ICP-based methods can fail due to the convergence to a local minimum.)

2. We convert the point cloud registration problem as an OPT (optimal partial transport) problem to estimate the correspondence between the source and target. The parts (red and blue) we plan to match are regarded as clean data. The part we will not match is regarded as "noise".
   In your task, most of the data points are noise data. It could affect the accuracy.

I also recommend you consider using d-dimension OPT for the step of correspondence estimation in each iteration. https://pythonot.github.io/_modules/ot/partial.html#partial_wasserstein. The code is easy to understand and you can modify it to improve the speed.

Btw, I have the code for the d-dimensional OPT solver, which is a little bit faster than this one, and am working on a library for OT (OPT) based point cloud registration. Will let you know when it can be public.

[ttsesm]

Hi @yikun-baio,

Many thanks for the response. Yes, you have more or less understood the problem correctly. The input though could be not only point clouds but also complete meshes including both vertices and faces, moreover it can include also multiple pieces, i.e. more than two. So you could have a multi-pair instead of a pair-wise problem, but for the simplicity of the problem let's consider the simpler case of a pair-wise only registration. You can have a look on the attached data ;-).

Yes, I do not have any scaling parameter to consider in my deformation function. I am only interested for the R|t estimation.

Regarding the challenged that you have mentioned: 1.I do not have any information about how much mass between the pieces is matched, so in principle the only information that I can use are the vertices, faces and color if it is available. Thus, I would need to rely mainly on geometric features. Having any prior knowledge of how probably the two pieces could be registered is also something unlikely to have. In principle, I start from the fact that the two pieces are initialized at the same original position and I am trying to find out whether they have any point where they can be assembled (There could also be the case that they might not).

2. Yes, the idea was that. Whether I could covert it as an OPT problem where in principle I could somehow match these partial parts. To be honest since my experience with the OPT is minimal I am not sure whether and how the "noise" would affect any OPT based solution.

Yes I was looking on different solutions either from POT, Geomloss or this recent work

Yes, I would be interested to check on the library that you are working with. Do you have any ETA? It would be interesting to check on any examples or tuts that you might already have with registration and d-dimensional OPT.

Thanks again.

bear.zip bottle.zip tombstone.zip