search

tusen-ai / SimpleTrack

MIT License
345 stars 64 forks source link

Evaluation result #10

Closed wenyiwangbst closed 2 years ago

wenyiwangbst commented 2 years ago

Hi, thank you for sharing your code. I have test run the code on the full waymo validation dataset and using the detection you send to me(thanks for doing that) and use waymo evaluation metrics to do the evaluation. The result I got shows as following: 2022-03-03 09-54-17 的屏幕截图 But in the paper the L2 Vehicle could achieve around 0.56 in MOTA which is about 3.5% higher than my implement. 2022-03-03 09-59-18 的屏幕截图 Can I ask if the released code is of the same setting in parameter as the one in the paper and what evaluation method are you guys using. Thanks again for all your work and help.

ziqipang commented 2 years ago

@wenyiwangbst Which config file did you use?

wenyiwangbst commented 2 years ago

Hi, thanks for replying. I'm using /waymo_configs/pd_kf_giou.yaml config file

wenyiwangbst commented 2 years ago

2022-03-07 09-41-03 的屏幕截图

ziqipang commented 2 years ago

@wenyiwangbst Please use vc_kf_giou for vehicles/cyclists and pd_kf_giou for pedestrians. I tested this repo on my side before open sourcing. If you still cannot re-produce, I will get back after eccv deadline.

BaiLiping commented 2 years ago

Tried the code on NuScenes, the result on evaluation set is AMOTA0.549. Also would it be possible to get your centerpoint inference result on Waymo? it took me a while to generate the inference result on NuScenes, and downloading the Waymo dataset is going to take a while. Thanks

BaiLiping commented 2 years ago

Hi, thank you for sharing your code. I have test run the code on the full waymo validation dataset and using the detection you send to me(thanks for doing that) and use waymo evaluation metrics to do the evaluation. The result I got shows as following: 2022-03-03 09-54-17 的屏幕截图 But in the paper the L2 Vehicle could achieve around 0.56 in MOTA which is about 3.5% higher than my implement. 2022-03-03 09-59-18 的屏幕截图 Can I ask if the released code is of the same setting in parameter as the one in the paper and what evaluation method are you guys using. Thanks again for all your work and help.

hi wenyi, any chance you can pass along that waymo inference file to me? my email is blp_engineer@outlook.com Thanks.

ziqipang commented 2 years ago

@BaiLiping @wenyiwangbst I just added the generated results at README. Please check out the dropbox links and see if they are helpful.

BaiLiping commented 2 years ago

Thanks a lot.