search

xhanxu / Mamba3D

[ACM MM 2024] Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Model
https://xhanxu.github.io/
81 stars 7 forks source link

Questions about reproducibility #9

Closed Nero-XD closed 3 months ago

Nero-XD commented 3 months ago
Hello, thank you for your excellent work, but when I started training Mamba3D from scratch, I achieved accuracies (w/o voting) of 91.91, 90.71, 91.15, and 91.77 on OBJ-BG, OBJ-ONLY, HARDEST, and MODELNET40, respectively. This is somewhat different from the 92.94, 92.08, 91.81, and 93.4 mentioned in your paper, especially on modelnet. May I ask if there are any other tips during training, and when can you release the pre training weights? obj_bg obj_only hardest modelnet
result in paper 92.94 92.08 91.81 93.4
my training result 91.91 90.71 91.15 91.77
xhanxu commented 3 months ago

Thank you for your interest in our work!

For the ScanObjectNN dataset, we have observed some variability in the results and are actively investigating the reasons. We suggest running the experiments multiple times, which should enable you to reach the results in the paper.

For the ModelNet40 dataset, please check if your data augmentation strategy aligns with the one used in the paper.

As for the pre-trained weights, we will try to release them within a few weeks. Please stay tuned!

Nero-XD commented 3 months ago

Thank you for your positive response. After checking, I found that the default data augmentation used in the code for training modelnet is rotation, while scale&transformer is used in the paper. I am preparing to retrain on the modelnet dataset. Looking forward to updates of your repository!

xhanxu commented 3 months ago

Looking forward to seeing your updated results here! We will improve the code to make it easier to reproduce the results.

Nero-XD commented 3 months ago

I'm back, after dozens of hours of training, I achieved a reproduction result of 93.72 on the ModelNet dataset. Your point cloud information processing work on MAMBA is really great, and I look forward to your future work!

xhanxu commented 3 months ago

We greatly appreciate your recognition of our work!

xhanxu commented 3 months ago

We have just released the pre-trained weights. Please check them out!