search

ch3cook-fdu / Vote2Cap-DETR

[CVPR 2023] Vote2Cap-DETR and [T-PAMI 2024] Vote2Cap-DETR++; A set-to-set perspective towards 3D Dense Captioning; State-of-the-Art 3D Dense Captioning methods
MIT License
76 stars 5 forks source link

Question about caption evaluation results #12

Closed TTXiann closed 3 months ago

TTXiann commented 3 months ago

i use the pretrain weight "scanrefer_scst_vote2cap_detr_pp_XYZ_RGB_NORMAL.pth" and get the follow result:

INFO: iou@0.5 matched proposals: [1537 / 2068], [BLEU-1] Mean: 0.6676, Max: 1.0000, Min: 0.0000 [BLEU-2] Mean: 0.5745, Max: 1.0000, Min: 0.0000 [BLEU-3] Mean: 0.4757, Max: 1.0000, Min: 0.0000 [BLEU-4] Mean: 0.3895, Max: 1.0000, Min: 0.0000 [CIDEr] Mean: 0.7525, Max: 6.3784, Min: 0.0000 [ROUGE-L] Mean: 0.5467, Max: 1.0000, Min: 0.1015 [METEOR] Mean: 0.2631, Max: 1.0000, Min: 0.0448

This result is not consistent with the paper, is this normal?

ch3cook-fdu commented 3 months ago

The result varies mainly because of: 1. The randomness in data pre-processing (point down sampling), and 2. Different PyTorch versions. Thus it is common that the performance may vary a little.

It is encouraged to execute all the training sessions.

TTXiann commented 3 months ago

The result varies mainly because of: 1. The randomness in data pre-processing (point down sampling), and 2. Different PyTorch versions. Thus it is common that the performance may vary a little.

It is encouraged to execute all the training sessions.

OK, I re-run the data preprocessing and got the following results：

INFO: iou@0.5 matched proposals: [1575 / 2068], [BLEU-1] Mean: 0.6838, Max: 1.0000, Min: 0.0000 [BLEU-2] Mean: 0.5880, Max: 1.0000, Min: 0.0000 [BLEU-3] Mean: 0.4855, Max: 1.0000, Min: 0.0000 [BLEU-4] Mean: 0.3970, Max: 1.0000, Min: 0.0000 [CIDEr] Mean: 0.7662, Max: 6.7565, Min: 0.0000 [ROUGE-L] Mean: 0.5550, Max: 1.0000, Min: 0.1015 [METEOR] Mean: 0.2660, Max: 1.0000, Min: 0.0448

This proves that randomness does exist. Thank you for your reply!