Speed Performance

[tchaton]

Dear @mrakotosaon,

I have been trying your pre-trained network to perform denoising on my pointcloud. The outlier removal works pretty well, however it is pretty slow.

What was the smallest architecture you tried which had good performance ?

Best, Thomas Chaton.

[mrakotosaon]

Dear @tchaton ,

Unfortunately we only have this architecture for the moment. Given the current performances, I think it should be possible to train a lighter network with decent accuracy.

[tchaton]

Dear @mrakotosaon, we tried with a smaller model and we got good outlier removal. However, the evaluation is still very slow as every point is evaluated. We have point clouds from 1M - 1B points, and PointCleanNet isn't well suited.

We started to investigate PoinNet++ with 2048 points and making predictions only in the inner ball (0.7 of the total patch, avg=1500 points at once). The points have been augmented with their local density, etc. The model doesn't seem to properly converge. However, the evaluation is at least 10-100 times faster.

Did you try anything in that direction?

Best, Thomas Chaton

[mrakotosaon]

Dear @tchaton ,

Sorry, since were more interested in having a local method for denoising, we do not have experiments with PointNet++.

Best,

Marie-Julie

[tchaton]

Hey @mrakotosaon,

We are going to opensource the code with PointNet ++. If you are interested, you could pursue with what we did.

The results looks like that. PointCleanNet++ with density performs slightly better than PointCleanNet (due to added density values) However, it is in average 30x times faster (which start to make it interesting for real applications).

F1 Score: 0.7652 [0.9455, 0.8437, 0.8914, 0.6498, 0.6229, 0.6383]
IoU:  0.6388 [0.8966, 0.7296, 0.8040, 0.4813, 0.4523, 0.4687]
Precision: 0.7398 [0.9906, 0.8264, 0.9238, 0.5843, 0.5404, 0.5731] (proportion of points classified as outliers that are outliers)
Recall: 0.8024 [0.9043, 0.8617, 0.8611, 0.7319, 0.7351, 0.7201] (proportion of outliers that are classified as outliers)
Accuracy: 0.8510 [0.9702, 0.9088, 0.9400, 0.7746, 0.7457, 0.7668]
32094.6667 Outliers classified as outliers
87046.8333 Clean points classified as clean
12953.1667 Clean points classified as outliers
7905.3333 Outliers missed
Total time: 555.4936 [572.1741, 558.4958, 565.1315, 552.2203, 542.7336, 542.2064]
Time in model: 368.7015 [380.0526, 369.9659, 375.6788, 367.9621, 359.4131, 359.1364]

pointcleannet on pointclouds with outliers

F1 Score: 0.8133 [0.9577, 0.9224, 0.9559, 0.7077, 0.6607, 0.6755]
IoU:  0.7069 [0.9188, 0.8560, 0.9156, 0.5476, 0.4933, 0.5100]
Precision: 0.8166 [0.9985, 0.9599, 0.9996, 0.6941, 0.6090, 0.6387] (proportion of points classified as outliers that are outliers)
Recall: 0.8140 [0.9201, 0.8878, 0.9159, 0.7218, 0.7219, 0.7167] (proportion of outliers that are classified as outliers)
Accuracy: 0.8885 [0.9768, 0.9573, 0.9759, 0.8296, 0.7881, 0.8032]
32561.0000 Outliers classified as outliers
91827.1667 Clean points classified as clean
8172.8333 Clean points classified as outliers
7439.0000 Outliers missed
Total time: 15.0990 [15.1408, 15.2529, 15.6364, 14.6221, 15.1170, 14.8249]
Time in model: 10.1432 [10.4114, 10.3767, 10.4668, 9.7787, 10.0393, 9.7866]

pointcleannet++ on pointclouds with outliers

Accuracy: 0.9341 [1.0000, 0.9971, 0.9964, 0.8426, 0.8934, 0.8754]
0.0000 Outliers classified as outliers
93414.5000 Clean points classified as clean
6585.5000 Clean points classified as outliers
0.0000 Outliers missed
Total time: 404.8740 [408.9886, 408.6170, 408.1409, 407.3212, 397.8924, 398.2841]
Time in model: 268.2906 [267.6145, 271.1312, 270.8507, 269.9953, 264.9703, 265.1814]

pointcleannet on pointclouds with no outliers

Accuracy: 0.3505 [0.3525, 0.3569, 0.1733, 0.4331, 0.3994, 0.3878]
0.0000 Outliers classified as outliers
35049.0000 Clean points classified as clean
64951.0000 Clean points classified as outliers
0.0000 Outliers missed
Total time: 8.4057 [11.3600, 7.7148, 7.4340, 8.6052, 7.7174, 7.6026]
Time in model: 4.6968 [4.5618, 4.7205, 4.5140, 5.2656, 4.5853, 4.5336]

pointcleannet++ on pointclouds with no outliers I think using normalized density means we're always going to detect some outliers, which is why we get so many false positives when there are no outliers

[mrakotosaon]

Hi @tchaton ,

Thank you for sharing these experiments with me, it is really interesting to see how it performs with PointNet++ and by evaluating more points at a time. If it corresponds to the data you are trying to denoise, using the density information can definitely be efficient! One of the reasons why we do not use density in our paper is also that we want to be able to adapt to any kind of noise: there could be some cases (when the noise is not gaussian and has some specific structure or if the sampling is not uniform) where the density information is not related to the way outliers should be classified.

Best,

Marie-Julie

[tchaton]

Dear @mrakotosaon,

From our observation, PointCleanNet worked pretty well for on data with non-gaussian noise. PointNet++ didn't seem to converge properly on when not provided the normalized density.

Best, Thomas Chaton

[ghost]

Dear @tchaton

This is code that does not implement the evaluation method. I would appreciate it if you could give me the code so that I can try what you have implemented. It would be very helpful.