search

Lilac-Lee / PointNetLK_Revisited

Implementation for our CVPR 2021 oral paper "PointNetLK Revisited".
MIT License
47 stars 5 forks source link

Question about reproducing the results in Table 1 #11

Open MickShen7558 opened 2 years ago

MickShen7558 commented 2 years ago

Hi,

First of all, thank you for your great paper and codes, and also the trained model you provided.

I trained the model on ModelNet40 using the default settings in your code twice. Then I found the performance on the evaluation set of my best model is much worse than the results in Table 1 in the paper. 

> ********************mean********************
> rot_MSE: 39.06458769078561, rot_RMSE: 6.250167013031381, rot_MAE: 1.0076209857277225, trans_MSE: 0.004946976434439421, trans_RMSE: 0.07033474743366241, trans_MAE: 0.009108548983931541
> ********************median********************
> rot_MSE: 8.32185003866275e-12, rot_RMSE: 2.884761695298721e-06, rot_MAE: 2.884761695298721e-06, trans_MSE: 3.552713678800501e-15, trans_RMSE: 5.960464477539063e-08, trans_MAE: 5.960464477539063e-08

The test was performed on the second half categories with 1000 points sampled. I also test your provided trained model with the same setting and got the same performance as in the paper:

********************mean********************
rot_MSE: 25.622303942241103, rot_RMSE: 5.061847878220078, rot_MAE: 0.9258663473932836, trans_MSE: 0.0031295332591980696, trans_RMSE: 0.05594222992658615, trans_MAE: 0.008093184791505337
********************median********************
rot_MSE: 4.499777552562683e-12, rot_RMSE: 2.121267911547875e-06, rot_MAE: 2.121267911547875e-06, trans_MSE: 1.9984014443252818e-15, trans_RMSE: 4.470348358154297e-08, trans_MAE: 4.470348358154297e-08

I guess this proves that adding point sampling during testing doesn't harm the performance?

Therefore, I would like to know more details about how you train and test over ModelNet40:

  1. Is the best model selected by total test loss or test transformation loss?
  2. In the paper you mentioned that

    We also partitioned 20% of the training set for evaluation. Does it mean that you used 20% training + the whole test set for validation during training and testing? Or did you only use the 20% training set for training and testing?

  3. Are there any additional settings for testing on ModelNet40? I see the code is for 3DMatch in default.

Thanks again.