search

slothfulxtx / cxtrack3d

CXTrack:Improving 3D Point Cloud Tracking with Contextual Information (CVPR2023)
MIT License
29 stars 3 forks source link

Cannot reproduce the results using pretrained weights #3

Closed AlexWang1900 closed 3 weeks ago

AlexWang1900 commented 1 year ago

I tested the cxtrack3d pretrained weights ,and I got: cxtrack_kitti_car_81.6_69.1.ckpt: ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ precesion │ 79.60032653808594 │ │ runtime │ 0.018718518316745758 │ │ success │ 67.48014688491821 │ └───────────────────────────┴───────────────────────────┘ cxtrack_kitti_pedestrian_91.5_67.0.ckpt: ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ precesion │ 87.14067840576172 │ │ runtime │ 0.019559158012270927 │ │ success │ 63.82514238357544 │ └───────────────────────────┴───────────────────────────┘

My environment : pytorch 2.01,cuda 11.8,pytorch_lightning 2.04 data are from kitti official site.

Under the same environment and data, M2TRACK (OPEN3DSOT) GOT:

github page results: M2Track-KITTI | Car | 67.43 | 81.04 | pretrained_models/mmtrack_kitti_car.ckpt

M2Track-KITTI | Pedestrian | 60.61 | 89.39 | pretrained_models/mmtrack_kitti_pedestrian.ckpt

my test results:

pedestrian: ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ precision/test_epoch │ 89.54376220703125 │ │ success/test_epoch │ 60.786376953125 │ └───────────────────────────┴───────────────────────────┘ car: ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ precision/test_epoch │ 81.04179382324219 │ │ success/test_epoch │ 67.47976684570312 │ └───────────────────────────┴───────────────────────────┘

StrivedTye commented 1 year ago

Hi. Using the pre-trained weights of cars, my reproduction results are 68.7531/81.2052 (succ/prec), also have small fluctuations.

pytorch 1.12.1, cuda 11.7, pytorch_lightning 1.5.10.

slothfulxtx commented 1 year ago

Hi, thanks for your interest. I've just test my pretrained weights of pedestrians on KITTI, here's my results, which is corresponding to our paper. image I'm trying to find out which factor leads to the fluctuations of performance.

slothfulxtx commented 1 year ago

I've just checked the precision and success metrics during the training process. As shown in the following figure, the performance metrics are still not stable when the model weights converge image I think this stems from unperfect testing data from KITTI and the nature of object tracking. image As long as the tracked target is missing in the point cloud sequences, the following predictions provided by the 3D tracker are unstable. This problem is partially solved by our recent work MBPTrack: Improving 3D Point Cloud Tracking with Memory Networks and Box Priors.

slothfulxtx commented 1 year ago

I suggest you can train CXTrack from scratch under your env to reproduce the similar performance.