search

OpenGVLab / LAMM

[NeurIPS 2023 Datasets and Benchmarks Track] LAMM: Multi-Modal Large Language Models and Applications as AI Agents
https://openlamm.github.io/
286 stars 15 forks source link

Zero-shot performances on ScanQA with Vicuna-7B #44

Closed UnderTheMangoTree closed 6 months ago

UnderTheMangoTree commented 9 months ago

Thanks for your work! I have used the official code with Vicuna-7B under 4 NVIDIA 3090. For the limitations of memory, i used the train_ds3.yaml, and changed the train_micro_batch_size_per_gpu to 1 and gradient_accumulation_steps to 16. However, the zero-shot performance of the trained model just 0.06570080862533692 on ScanQA. It is far lower than LAMM with vicuna-13, 26.54, which you published in the paper.

I would like to ask the performance of LAMM with Vicuna-7B you tested on various 3D_Benchmark, and the details of training process. Thank you very much 😁!

wangjiongw commented 9 months ago

Thanks for your intereset and question. For your question, we re-trained a vicuna7b version for 3D data this week and its performance on ScanQA is 21.39, which is slightly lower than that of 13B version and somehow reasonable. How about your results on other tasks and datasets? Maybe you can give us more details.

wangjiongw commented 6 months ago

I will close this issue due to long-term inactivation. Please reopen it if necessary.