Loss calculation did not involve shape_x

[ihmc3jn09hk]

Hi @pvnieo , thank you for the update of the repo.

I am not sure whether it is a mistake on the code or my understanding. Should the loss involve the difference to shape_x's geodesics distance ?

e.g.

loss = torch.sqrt(((P * geodesic_dist_y - geodesic_dist_x ) ** 2).sum((1, 2)))

In the updated code, the calculation does not involve any information towards shape_x.

https://github.com/pvnieo/FMNet-pytorch/blob/116b48ecc44781db1de87ea1089398e8754a3bb7/fmnet/loss.py#L23

[pvnieo]

Hi @ihmc3jn09hk The formula you suggested will not work, since the shapes X and Y doesn't not always have the same number of vertices. So for example, if shape X has 1000 vertices and shape Y has 1010 vertices, their geodesic matrix will have the shape 1000x1000 and 1010x1010 respectively, and the geodesic_dist_y - geodesic_dist_x will rise an error.

Also, in the paper, the Matrix P is a soft assignment matrix, meaning it transfer from shape 1 to shape 2, and so to compute the error, we are only interested in shape 2, to verify if the transfer worked well. Note that when generating batches, we generate the couple (x,y) and also (y,x), so we will always check in the two directions.

Hope this answer your question!

[ihmc3jn09hk]

Hi @ihmc3jn09hk The formula you suggested will not work, since the shapes X and Y doesn't not always have the same number of vertices. So for example, if shape X has 1000 vertices and shape Y has 1010 vertices, their geodesic matrix will have the shape 1000x1000 and 1010x1010 respectively, and the geodesic_dist_y - geodesic_dist_x will rise an error.

Also, in the paper, the Matrix P is a soft assignment matrix, meaning it transfer from shape 1 to shape 2, and so to compute the error, we are only interested in shape 2, to verify if the transfer worked well. Note that when generating batches, we generate the couple (x,y) and also (y,x), so we will always check in the two directions.

Hope this answer your question!

Thanks for the answer @pvnieo . I get what you mean. I was looking at exactly eq. 5 from the paper. I don't quite understand why minimizing P*Dy if Dy ONLY means the geodesic distances of every points on y. Since it's never 0 and P is the map P: X_feature ->Y_feature, then it will requires C to be 0 since Phi and Psi are also non-zero. So my guess will be Dy means the difference between the geodesic_y and P*geodesic_x. Also, I thought the sampling will garantee the number of vertices/features be identical. The paper is abit complicated to me and I am quite confused at some area, like eq. 5.

Moreover, is the P returned from FMNet.forward() the point-to-point correspondence from Y->X ? I copied that part to SURFNet and tried to extract the max indices from it, e.g. P=torch.max(P, dim=1), for inference in the SURFNet but it seems not correct (the weights I used was simply trained with faust_remesh at 20 epochs and batch-size=4). Or

1. I have to train it longer?
2. I have to use a KDTree to find the closest features instead?

https://github.com/pvnieo/FMNet-pytorch/blob/116b48ecc44781db1de87ea1089398e8754a3bb7/fmnet/model.py#L90

Sorry the later part is related to the SURFNet and I should have asked in that repo. Maybe I should upload the code I wrote for inference in SURFNet for better clarification. Or do you mind if I send you through email ?

[pvnieo]

P is a soft assignment, you can use it to produce figures like in figure 2.

D_y is the geodesic matrix of shape Y and not the difference, it's written in the figure I sent.

To find the point to point map, please convert the functional map produced by the networks using KDTree for example. If the result are not good enough, train longer, until you hit the limit of the network.

To see if everything is working for you, try to overfit a small dataset, and test on it!

[ihmc3jn09hk]

Lets close the issue first. @pvnieo Thank you very much

[ihmc3jn09hk]

I know it is not necessary but hope you may give me some help on inference of SURFNet. I have attached code I tried in the link including trained weights, e.g. overfit as you suggested for testing (4 geometries, n-vertices sample 2500, batch 4, epochs 10000) with loss around 54~.

The evaluation code, trained weights

...
#epoch:9997, #batch:3, #iteration:29991, loss:59.408660888671875
#epoch:9998, #batch:1, #iteration:29992, loss:65.03675842285156
#epoch:9998, #batch:2, #iteration:29993, loss:59.46687316894531
#epoch:9998, #batch:3, #iteration:29994, loss:62.42640686035156
#epoch:9999, #batch:1, #iteration:29995, loss:77.87380981445312
#epoch:9999, #batch:2, #iteration:29996, loss:74.61470031738281
#epoch:9999, #batch:3, #iteration:29997, loss:58.388031005859375
#epoch:10000, #batch:1, #iteration:29998, loss:71.05875396728516
#epoch:10000, #batch:2, #iteration:29999, loss:64.3044662475586
#epoch:10000, #batch:3, #iteration:30000, loss:54.037261962890625

From the paper and your suggestion, it says P is a soft correspondence matrix, which can be interpreted as the probability of point x∈ X mapping to point y∈Y (see Figure2);

So my guess will be the point-to-point (p2p) correspondence can be extracted by finding the max. value along each row of P to form a p2p vector. The following is a snap shot I implemented. But it seems to be wrong because no KDTree is needed.

    p2p = torch.argmax(P, dim=1)
    print( p2p )

I tried to refer from the pyFM repo. you suggested in another Thread, I have a hard time to correlate between the implementations.

tree = KDTree(eigvects1[:, :k1] @ FM.T)  # Tree on (n1,k2)
matches = tree.query(eigvects2[:, :k2], k=1, return_distance=False).flatten()  # (n2,)

Could you please help ?

[pvnieo]

in the pyFM repo, FM correspond to the functional map C, .T is the transpose of the matrix, eigvects are the eigen vectors which are evecs in this implementation! k1 and k2 are the number of eigenvectors chosen, must be equal and I suggest you use the same number as in training

[ihmc3jn09hk]

in the pyFM repo, FM correspond to the functional map C, .T is the transpose of the matrix, eigvects are the eigen vectors which are evecs in this implementation! k1 and k2 are the number of eigenvectors chosen, must be equal and I suggest you use the same number as in training

Ok, let me re-train it with the same number of eigen-vectors/basis. I was using the default 40 from the SURFNet repo while inferencing at 120. Moreover, does the n-vertices/samples matters ? From the paper, it said smaller amount of vertices, e.g. 15000 comparing with full resolution, can lead to better results. Yet the re-meshed database contain meshes with ~5K vertices, so I should try maybe 4000K vertices for training as well.