search

Pointcept / GPT4Point

[CVPR'24 Highlight] GPT4Point: A Unified Framework for Point-Language Understanding and Generation.
https://gpt4point.github.io/
MIT License
304 stars 19 forks source link

Can't reproduce #25

Closed zoesoon closed 1 month ago

zoesoon commented 1 month ago

Hi, We used the point_encoder_pointbert_wcolor.pth and gpt4point_pretrain_stage2_opt2.7b.pth files provided on Huggingface to evaluate, but we could not reproduce the accuracy reported in your paper. Here are the accuracy metrics we obtained:

Bleu_1: 0.135
Bleu_2: 0.072
Bleu_3: 0.043
Bleu_4: 0.028
METEOR: 0.065
ROUGE_L: 0.161
CIDEr: 0.306

And the accuracy metrics after local training and evaluation are:

Bleu_1: 0.024
Bleu_2: 0.004
Bleu_3: 0.000
Bleu_4: 0.000
METEOR: 0.007
ROUGE_L: 0.034
CIDEr: 0.004

Could you please explain why this discrepancy occurs? Additionally, will you be uploading other evaluation tasks including zero-shot learning and robot dialogue on GitHub?

Thanks!!

Qi-Zhangyang commented 1 month ago

Is there some error in the code, I will update GPT4Point++ code in Aug 8th.