script generates lower numbers than those in Table.9 (linear SVM on learned embeddings)

[cxlcl]

Hi,

On ModelNet40, I was trying to reproduce the experiment results in Table.9 in the supplementary, but got lower numbers: PointNet with Jigsaw task: 82% vs. 87.5% in the paper PointNet with OcCo task : 85.49% vs. 88.7% in the paper

Similar case for the DGCNN backbone

The scripts to learn the embeddings are copied from provided bash template, e.g: python train_completion.py --gpu 0 --dataset modelnet --model pointnet_occo --log_dir modelnet_pointnet_vanilla for PointNet with OcCo task. For linear SVM training and testing, the scripts are also carried from the bash template, e.g: python train_svm.py --gpu 0 --model pointnet_util --dataset modelnet40 --restore_path log/completion/modelnet_pointnet_vanilla/checkpoints/best_model.pth

So I'm wondering if there is some more hyper-parameter tuning needed but not provided in the templates.

Thanks!

[hansen7]

Hi @cxlcl , thanks for pointing out!

I just uploaded the pre-trained models that are the best for classification, please see the google drive here :)

Also, maybe, see the this jupyter notebook (in pdf) for how I get the reported numbers in the appendix, same to train_svm.py, it is higher than your reported number (85.49%), I remember 88.7% is the best results in my hand after tons of runs.

For the Jigsaw task, I think I do not provide the pre-trained weights on that, right?

It is worth noting that even for the same checkpoints, the performance of the linear SVM still varies due the batch normalisation with shuffle. Feel free to pop up if your reproduced results are far away from the reported numbers

[cxlcl]

Thanks a lot for providing the pre-trained models!

As for the Jigsaw task, I tried to train from scratch using the provided batch template. Then test with train_svm.py. Same goes for all other models. That might be another reason why my reported number is lower, besides not running train_svm.py multiple times - I just ran once with only one checkpoint (the last one).

[cxlcl]

I've tested with the provided weights (PointNet trained with OcCo) and got ~86.3% accuracy - higher than the one I trained from scratch. Expect to get higher number with more runs.