On the error of the benchmark on Stanford Lounge data

[hongtaowu67]

Hi Bing Jian,

I saw that when running on the 295 pairs of the Stanford lounge dataset, the claim is " The avg pair-wise pose difference among these 295 pairs is ~4.84 degree." However, in the chart underneath, the "Rotation angle error (in degrees)" is 0.96. I am wondering which one is correct?

Thanks!

[bing-jian]

IIRC, these two are different things. The pair-wise pose differences are distance between ground truth poses, indicating how far away the two poses in each pair. The rotation angle error is the difference between the registration result and the ground truth relative pose, indicating how accurate is the registration result.

[hongtaowu67]

Thank you! I think it is clear. I am closing this issue.

[hongtaowu67]

I am trying to replicate the result from section 6.1.2 Experiments with Range Data in the paper. I am running on the dragon_stand dataset for 24, 48, 72, and 96 degrees pose difference (TABLE 2 in the paper) using the parameter provided in dragon_stand.ini. I cannot get the result reported in the paper. I am wondering did you downsample the dragon stand data in this experiment? Are you using the the method here to do the downsampling? Thanks!

[hongtaowu67]

I forgot to reopen the issue. Reopening with this comment.

[bing-jian]

I am trying to replicate the result from section 6.1.2 Experiments with Range Data in the paper. I am running on the dragon_stand dataset for 24, 48, 72, and 96 degrees pose difference (TABLE 2 in the paper) using the parameter provided in dragon_stand.ini. I cannot get the result reported in the paper. I am wondering did you downsample the dragon stand data in this experiment? Are you using the the method here to do the downsampling? Thanks!

Hongtao, are you able to run the code below https://github.com/bing-jian/gmmreg/blob/master/expts/dragon_expts.py

[hongtaowu67]

I am trying to replicate the result from section 6.1.2 Experiments with Range Data in the paper. I am running on the dragon_stand dataset for 24, 48, 72, and 96 degrees pose difference (TABLE 2 in the paper) using the parameter provided in dragon_stand.ini. I cannot get the result reported in the paper. I am wondering did you downsample the dragon stand data in this experiment? Are you using the the method here to do the downsampling? Thanks!

Hongtao, are you able to run the code below https://github.com/bing-jian/gmmreg/blob/master/expts/dragon_expts.py

Yes. I am able to run the code. And the result is actually better than the result reported in the paper: I got 30/30 success rate; the paper reports 29/30 success rate. It is the result for a 24-degree pose difference. However, I only got 17/30 and 6/30 when the pose difference is 48 degrees and 72 degrees, respectively.

I have attached the data. This data is generated by the code itself, i.e., downsampling using the downsample function in common_utils.py dragon_stand_down_gmmreg.tar.gz

[bing-jian]

Hongtao, thanks for testing and letting me know results you got. As you can tell from the git history, both the underlying implementation and the parameter setting used by current version of https://github.com/bing-jian/gmmreg/blob/master/expts/dragon_expts.py won't be the same as those being used when the paper was prepared more than 10 years ago. :-). In general, there are a few ways to improve the registration accuracy of gmmreg algorithm: 1) apply normalization on the point sets first 2) try multi-scale registration 3) use better data downsampling and select the point set with fewer points as the moving dataset. Looks like the current version of https://github.com/bing-jian/gmmreg/blob/master/expts/dragon_expts.py (updated in June 2019) is not compatible with Python3 and latest version of Open3D (for visualization only), I will upload a new version later. Since you were able to run the dragon_expts.py, I guess you made some local changes to make that script work, for now, can you please try the following configuration (other parts in dragon_stand.ini should remain unchanged) [GMMREG_OPT] normalize = 1 max_function_evals = 100 100 100 100 sigma = .5 .2 .1 .05 level = 2

[hongtaowu67]

Thanks, Bing! I understand that the paper is from a decade ago and the details of the experiment may be hard to recall. I was asking because I think I probably had something done incorrectly. And your advice actually helps!

I re-downsampled the data with fixed-size voxel grid. With the parameter you provided, I can now get 30/30, 20/30, 13/30 success rate for 24, 48, and 72 degree pose difference. It is a big improvement compared to the my previous trial. This actually matched with the results you provided in the TPAMI paper. I am closing this issue. Thank you!

[bing-jian]

Hongtao, thanks for trying and letting me know the results.

BTW, I updated (and refactored) https://github.com/bing-jian/gmmreg/blob/master/expts/dragon_expts.py to make it work with Python 3 and latest version of Open3D. (The gmmreg binary also need to be updated with the latest C++ code due to a typo fix in config file). The results I got now are 30/30, 22/30, 16/30 with the fixed-size voxel grid downsampling, the new parameters and always using the smaller point set as the moving point set.

[hongtaowu67]

I am sure about what you mean by "always using the smaller point set as the moving point set". The 30 pairs for registration are 0-24, 24-48, ..., and 24-0, 48-24, ... right? Let's say 0.ply has less points then 24.ply. If I understand correctly on "always use the smaller point set as the moving point set", then 24-0 and 0-24 will be equivalent because in both case 0 will be the moving point set.

[bing-jian]

That's correct and we just take the inverse of the resulting transformation for those pairs that got switched. And that's why the numbers of success got from this way are all even numbers.