Table 3 is invalid

[chrischoy]

Hi Xingyu,

The table 3 is flawed.

You used a different split than the one used in the Minkowski Net paper. The split I gave you was not the final one that was used in the CVPR official paper and it should be modified accordingly to report the numbers accurately. This was a mistake on my part since I did not follow up with you and didn't release the code on time, but please update the numbers with the correct split. I've never seen any semantic segmentation dataset that PointNet++ beats Minkowski Net and this will give false signal to the community.

The reason I created another split was that the scenes are too similar and MinkowskiNet beats the dataset too easily, so I created a new split.

I'll report the number on the easy one as well.

For those who are not familiar with 3D semantic segmentation.

Please go to the ScanNet benchmark. http://kaldir.vc.in.tum.de/scannet_benchmark/ Basically, PointNet++, which the Table 3 in the paper claimed to outperform MinkowskiNet, is pretty down on the list. ScanNet is the most widely used benchmark for 3D semantic segmentation.

[chrischoy]

I uploaded the CVPR19 split https://github.com/chrischoy/SpatioTemporalSegmentation/tree/master/splits/synthia4d.

Please use the one marked *cvpr19.txt for comparisons.

[chrischoy]

Just to follow up on the thread. We decided it would be best to revisit this issue after the imminent deadlines.

[xingyul]

Hi Chris,

Thanks for posting this issue.

Our experiments on Synthia followed the description of train/val/test split in your CVPR 2019 paper. For example, the test split has 1886 sequences, which corresponds to the total number of 3D scenes in sequence 06-SUNSET and 06-SPRING, as described in your paper. The new split files in the link you shared on Nov 2 are different from what your CVPR 2019 paper describes.

For now, the best we can do is to follow what's already been officially published. We would appreciate it if you could first fix the discrepancy between your CVPR 2019 paper and the new split files you shared, e.g. by updating your arXiv paper.

The only mistake we made is in the last column of Table 3 where we reported overall accuracy instead of mean accuracy. But this mistake has nothing to do with the dataset split. We have updated our paper on arXiv to fix it.

A few more comments regarding the performance of PointNet++ on the ScanNet leaderboard: first, the submission was not from the original author; second, for each dataset, the best possible hyperparameters of PointNet++ are different, implying that the hyperparameters need to be carefully tuned. Therefore, we can't draw conclusions about the performance of PointNet++ and MinkowskiNet on the Synthia dataset simply from the rankings on the ScanNet leaderboard.

Additionally to semantic segmentation, we have shown that the general idea of MeteorNet also works on other vision problems such as Scene Flow and activity recognition.

[chrischoy]

I also retrained the MinkNet on CVPR19 split as well.

I haven't trained the 4D network, but with 3D MinkNet18, we got CVPR19 test 82.762% mIoU, no rotation average, no sliding window outperforming all MeteorNet models significantly.

The training script is available on https://github.com/chrischoy/SpatioTemporalSegmentation

I'll update the performance of the 4D network as well on.