Checkpoint performance is inconsistent with paper

[Coolshanlan]

Hello author,

After downloading the checkpoint provided by you, I re-ran test.py and found a discrepancy between the performance calculated and that presented in the paper. Below is the performance at 1% bbox (1000):

Have I missed any details? How can I achieve the performance mentioned in the paper? Thanks!

[Luoyadan]

Hi Coolshanlan,

I have checked Table 2 and it seems consistent with your reproduced results. May I know which particular issue you refer to? We used wandb to merge three trails and there might be a very tiny difference as the merging strategy in wandb is different from sklearn plots.

Cheers, Yadan

[Coolshanlan]

Thank you very much for your reply. I see, indeed it matches with table 2, but why are the scores in table 1 EASY 80.7, MOD 67.81, HARD 62.81? What is the difference between table 1 and table 2? I thought 1% of bbox is approximately 1000.

[Luoyadan]

Hi,

1% bbox should be around 1279.

Cheers, Yadan

[Coolshanlan]

Hello, thank you for your response. How to calculate that 1% corresponds to 1279 bboxes? Because in the training log it shows Car(14357) + Pedestrian(2207) + Cyclist(734) -> total = 17298.

2024-04-08 15:43:58,892   INFO  Total samples for KITTI dataset: 3712
2024-04-08 15:43:58,943   INFO  Database filter by min points Car: 14357 => 13532
2024-04-08 15:43:58,943   INFO  Database filter by min points Pedestrian: 2207 => 2168
2024-04-08 15:43:58,944   INFO  Database filter by min points Cyclist: 734 => 705
2024-04-08 15:43:58,952   INFO  Database filter by difficulty Car: 13532 => 10759
2024-04-08 15:43:58,954   INFO  Database filter by difficulty Pedestrian: 2168 => 2075
2024-04-08 15:43:58,954   INFO  Database filter by difficulty Cyclist: 705 => 581

[Luoyadan]

Hi CoolShanlan,

The 17298 is the total number of bbox in the training set. We calculate 1% as the size of unlabeled data pool ( total # - # bbox in the randomly selected initial set). Sorry for the confusion caused.

Cheers, Yadan

[Coolshanlan]

Hello, thank you for your response. I'm sorry, I still don't quite understand. I thought 3712 frames were all the data (including labeled and unlabeled), and there are a total of 17298 bounding boxes in those 3712 frames. Are you saying the unlabeled data pool consists of more than just these 3712 frames?