Cannot reproduce results on KITTI pedestrian

[slothfulxtx]

Hi, Thanks for your contribution to 3D SOT research area. However, I met some problems when I used your code. It's so weird that I can reproduce the result of KITTI car while I cannot reproduce the result of KITTI pedestrian using pretrained models.

CUDA_VISIBLE_DEVICES=2,3 python main.py --gpu 0 1 --cfg cfgs/BAT_Car.yaml  --checkpoint ./pretrained_models/bat_kitti_car.ckpt --test
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2,3]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [2,3]
Validating: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [02:55<00:00,  1.43s/it]-
--------------------------------------------------------------------------------
DATALOADER:0 VALIDATE RESULTS
{'precision/test': 78.87921142578125,
 'precision/test_epoch': 78.87921142578125,
 'success/test': 65.34402370452881,
 'success/test_epoch': 65.34402370452881}

CUDA_VISIBLE_DEVICES=2,3 python main.py --gpu 0 1 --cfg cfgs/BAT_Pedestrian.yaml  --checkpoint ./pretrained_models/bat_kitti_pedestrian.ckpt --test                                                                                                                                                                                                                                  
GPU available: True, used: True                                                                                                                                                             
TPU available: False, using: 0 TPU cores                                                                                                                                                    
IPU available: False, using: 0 IPUs                                                                                                                                                         
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2                                                                                                                                       
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2                                                                                                                                       
----------------------------------------------------------------------------------------------------                                                                                        
distributed_backend=nccl                                                                                                                                                                    
All distributed processes registered. Starting with 2 processes                                                                                                                             
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2,3]
Validating: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [03:01<00:00,  3.21s/it]
--------------------------------------------------------------------------------
DATALOADER:0 VALIDATE RESULTS
{'precision/test': 71.10257720947266,
 'precision/test_epoch': 71.10257720947266,
 'success/test': 43.36358308792114,
 'success/test_epoch': 43.36358308792114}
--------------------------------------------------------------------------------

I evaluate BAT model with cuda10.2 and pytorch 1.8, thus I suppose this phenomenon is not caused by the platform. Do you also meet with such a phenomenon? Look forward to your explanation.

Thanks~

[Ghostish]

This is due to a common trick that is used in most appearance-matching-based approaches. This trick helps to handle distractors but brings uncertainty in the testing. Our motion-centric approach M^2-Track does not have this issue and the code will be released soon. Stay tuned.

[Ghostish]

Duplicate of #9

[slothfulxtx]

Thanks for your reply, it's really helpful!