Questions on few-shot baseline

[Faye0123]

Hi! Thanks for the great work! I have a question regarding the baseline results of few-shot classification. How do you get the results of "PointNet rand" in Table 2? Did you train PointNet from scratch using cross entropy loss on the sampled K-way N-shot training data? Look forward to your reply! Thanks!

[hansen7]

Thanks for your interest :) Sadly we have to adapt the very abnormal setting (model + training) in the original paper [1] to make a fair comparison.

[1] Self-Supervised Few-Shot Learning on Point Clouds, NeurIPS 2020

[Faye0123]

Thanks for your quick reply! This is a bit strange. Seems they train the baseline model from scratch and then take the embeddings to train an additional SVM to report the results. Did I get it right?

As for your settings, may I confirm with you that you use the unsupervised model pre-trained on the whole ModelNet40 training set and then take the encoder to extract features and train a linear SVM on the sampled K-way N-shot data (for both ModelNet40 and ScanObjNN in Table 2)? For testing, did you use all the samples from the K classes in the test set? Thank you very much!

[hansen7]

Thanks for your quick reply! This is a bit strange. Seems they train the baseline model from scratch and then take the embeddings to train an additional SVM to report the results. Did I get it right?

emm, not exactly. In section 3.1: "For the unsupervised methods (3DGAN, Latent-GAN, PointCapsNet, and FoldingNet), we train their network and assess the quality of their final embeddings on a classification task using linear SVM as a classifier." As they don't describe clearly in the paper. You can refer to their code:

https://github.com/charusharma1991/SSL_PointClouds/blob/6de7e38df444b472e134a7d4c80ad540fd810db0/Classification/train_classifier.py#L47

https://github.com/charusharma1991/SSL_PointClouds/blob/6de7e38df444b472e134a7d4c80ad540fd810db0/Classification/train_classifier.py#L83-L99

https://github.com/charusharma1991/SSL_PointClouds/blob/6de7e38df444b472e134a7d4c80ad540fd810db0/Classification/train_classifier.py#L83-L99

So they still end-to-end train the Feature Encoder +MLPs for the few-shot learning.

As for your settings, may I confirm with you that you use the unsupervised model pre-trained on the whole ModelNet40 training set and then take the encoder to extract features and train a linear SVM on the sampled K-way N-shot data (for both ModelNet40 and ScanObjNN in Table 2)? For testing, did you use all the samples from the K classes in the test set?

We train the same way, with MLPs for output layers. And for the test data, it is the same K classes, but with a fixed number of samples (from test split) for each category, same as here:

https://github.com/charusharma1991/SSL_PointClouds/blob/6de7e38df444b472e134a7d4c80ad540fd810db0/Classification/data_classifier.py#L17

Thank you very much!

No worries, hope it helps :)

[hansen7]

Thanks for reminding me, I will migrate the code to this repo before the conference, currently, it is a bit incomplete.

[Faye0123]

Great!

"So they still end-to-end train the Feature Encoder + MLPs for the few-shot learning." Did you try to train this model (PointNet Rand) on your own? Why I am getting much better results than the reported numbers...

Another question is the standard deviation of the 10 runs. I'm getting quite large stds since in each run different categories and data samples are sampled. Any hints on this?

Thank you very much for your help!

[hansen7]

Based on their setting, right?

[Faye0123]

Based on their setting, right?

yes

[hansen7]

I found the same issue。。。but the reviewer will ask why not follow the numbers in previous work, it is really annoying = =

[Faye0123]

I found the same issue。。。but the reviewer will ask why not follow the numbers in previous work, it is really annoying = =

I see... Any comments on the std issue?

[hansen7]

it is, especially when the number of class/samples are small, we repeated N=10 runs and report the standard error (ste/\sqrt{N})

[Faye0123]

it is, especially when the number of class/samples are small, we repeated N=10 runs and report the standard error (ste/\sqrt{N})

I see. It is the standard error of the mean, which is std / \sqrt{N}. Thank you very much!

[hansen7]

no worries, hope it helps :)

[Faye0123]

Hi!

Sorry to bother you again. I just want to confirm with you that did you use the whole ModelNet40 training set for all your unsupervised pre-training, especially few-shot classification in Table 2? Thanks!

[hansen7]

Yes, we do :)

[Faye0123]

Many thanks!

[Faye0123]

Hi! Me again:) I wonder how you do the finetuning on S3DIS using the pretrained model on ModelNet40, since S3DIS has 9-dimensional features while ModelNet only has 3-dimensional coordinate features. Looking forward to your reply! Thanks!

[hansen7]

https://github.com/hansen7/OcCo/blob/c218a2bb446f91702cf8fa6f56bb3a1da406009f/OcCo_Torch/utils/Torch_Utility.py#L17-L34

[Faye0123]

Thank you very much!