Questions on performance

[StanLei52]

Hi author,

Thank you for working on the great OpenShape. I have some questions on the performance of the models.

1. I trained the PointBert on this codebase by running python3 src/main.py --trial_name pointbert_all model.name=PointBERT model.scaling=4 model.use_dense=True training.lr=0.0005 training.lr_decay_rate=0.967. I assumed this is the case without hard negative mining for PointBERT and found that the performance is less promising than using hard negative mining, e.g., -4% on Objaverse LVIS. May I know the performance of PointBERT_all w/o hard negative mining so that I can get a sense if I reproduced the correct results?
2. For Table 1 and Table 2 in the paper, how do you report the results? I noticed that there is std.dev. in Tab 1 but no such stat for Tab 2.
3. Regarding to Q1, I'd like to know the effect of hard negative mining for PointBERT under the three benchmarks since Tab 3 in the paper is for SparseConv.

Thanks!

[Colin97]

1. Yes, this is the case without hard negative mining. To add hard negative mining for PointBERT, you need to first train a model using random batch sampling until convergence and then fine-tune the model with hard negative mining batches. Also, the KNN indices we provided are for SparseConv, and you may want to create one for PointBERT using the trained model from the first stage.
2. Note that the std. in Tab 1 is calculated across different backbones and is used to demonstrate the performance gap between different backbone models. In Table 2, we don't need to compare the performance gap.
3. We have yet to test it for PointBERT and may add it later.

[StanLei52]

Thank you so much for your reply. Based on your reply for 1 and 3, is the config for PointBERT in Tab 1 w/o hard negative mining?

[Colin97]

yes

[StanLei52]

OK. May I confirm that python3 src/main.py --trial_name pointbert_all model.name=PointBERT model.scaling=4 model.use_dense=True training.lr=0.0005 training.lr_decay_rate=0.967 can reproduce the PointBERT perf as shown in Tab 1? I did this with 8-V100 and got worse results on Objaverse(~42%) and ScanObjectNN(~52%).

[eliphatfs]

Thank you for reporting the issue. We just noticed that a patch was missing from the released code. The point cloud is not randomly permuted during training, which leads to bias in PointBERT tokenization -- the FPS grouper takes points with largest indices for each centroid. I just did a quick blind fix in commit 4748adb8ad202bc2446fddd0eee5cc1b32f13f3f.

Another difference may be that we trained the model with 4 A100s, each consuming a batch size of 48 resulting in a total batch size of 196. The effective batch size may be different if you have 8 V100s.

[StanLei52]

Thank you for your update, I will retry with this version.

[StanLei52]

Hi @eliphatfs, before I get started, I still have some questions on the new commit.

Compared to the previous version, the random sample is updated for get_others for Four Dataset, getitem for ModelNet40Test and getitem for ObjaverseLVIS. Is this correction enough for pretraining on Four dataset, since get_objaverse for Four is NOT updated in your last commit? Could you please double-check for this?

Thanks!

[eliphatfs]

Thank you for pointing out it and sorry, I missed that.