search

OpenDriveLab / UniAD

[CVPR'23 Best Paper Award] Planning-oriented Autonomous Driving
Apache License 2.0
3.11k stars 335 forks source link

Evaluation about uniad_base_e2e.pth #179

Closed fsong666 closed 3 months ago

fsong666 commented 3 months ago

load checkpoint from local path: ./ckpts/uniad_base_e2e.pth load checkpoint from local path: ./ckpts/uniad_base_e2e.pth 2024-04-09 19:00:24,742 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.0.conv2 is upgraded to version 2. 2024-04-09 19:00:24,748 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.1.conv2 is upgraded to version 2. 2024-04-09 19:00:24,752 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.2.conv2 is upgraded to version 2. 2024-04-09 19:00:24,757 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.3.conv2 is upgraded to version 2. 2024-04-09 19:00:24,761 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.4.conv2 is upgraded to version 2. 2024-04-09 19:00:24,765 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.5.conv2 is upgraded to version 2. 2024-04-09 19:00:24,770 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.6.conv2 is upgraded to version 2. 2024-04-09 19:00:24,774 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.7.conv2 is upgraded to version 2. 2024-04-09 19:00:24,779 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.8.conv2 is upgraded to version 2. 2024-04-09 19:00:24,783 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.9.conv2 is upgraded to version 2. 2024-04-09 19:00:24,788 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.10.conv2 is upgraded to version 2. 2024-04-09 19:00:24,792 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.11.conv2 is upgraded to version 2. 2024-04-09 19:00:24,796 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.12.conv2 is upgraded to version 2. 2024-04-09 19:00:24,801 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.13.conv2 is upgraded to version 2. 2024-04-09 19:00:24,805 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.14.conv2 is upgraded to version 2. 2024-04-09 19:00:24,810 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.15.conv2 is upgraded to version 2. 2024-04-09 19:00:24,814 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.16.conv2 is upgraded to version 2. 2024-04-09 19:00:24,819 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.17.conv2 is upgraded to version 2. 2024-04-09 19:00:24,823 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.18.conv2 is upgraded to version 2. 2024-04-09 19:00:24,827 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.19.conv2 is upgraded to version 2. 2024-04-09 19:00:24,832 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.20.conv2 is upgraded to version 2. 2024-04-09 19:00:24,836 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.21.conv2 is upgraded to version 2. 2024-04-09 19:00:24,841 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.22.conv2 is upgraded to version 2. 2024-04-09 19:00:24,846 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.0.conv2 is upgraded to version 2. 2024-04-09 19:00:24,856 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.1.conv2 is upgraded to version 2. 2024-04-09 19:00:24,863 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.2.conv2 is upgraded to version 2. The model and loaded state dict do not match exactly

unexpected key in source state_dict: bbox_size_fc.weight, bbox_size_fc.bias, pts_bbox_head.query_embedding.weight, pts_bbox_head.transformer.reference_points.weight, pts_bbox_head.transformer.reference_points.bias

[>>>>>>>>>>>>>>>>>>>>>>>>>>] 6020/6019, 2.3 task/s, elapsed: 2571s, ETA: 0s writing results to output/results.pkl Occ-flow Val Results: iou 64.2 & 40.5 pq 56.3 & 34.7 sq 76.2 & 71.3 rq 73.9 & 48.7 num occ evaluated:2558 ratio occ evaluated: 42.5% +-------------+--------+--------+--------+---------+---------+---------+ | metrics | 0.5s | 1.0s | 1.5s | 2.0s | 2.5s | 3.0s | +-------------+--------+--------+--------+---------+---------+---------+ | obj_col | 0.0000 | 0.0002 | 0.0008 | 0.0100 | 0.0126 | 0.0086 | | obj_box_col | 0.0002 | 0.0003 | 0.0133 | 0.0387 | 0.0580 | 0.0684 | | L2 | 0.3493 | 2.9128 | 7.9545 | 15.1740 | 24.4322 | 35.4346 | +-------------+--------+--------+--------+---------+---------+---------+

I loaded your trained uniad_base_e2e.path to eval, its result: L2: 1s: 2.9128 2s:15.1740 3s:35.4346 Col.Rate%: 1s of obj_box_col 0.03%; 2s:3.87% 3s:6.84%.

This is different from your paper: L2: 1s: 0.48 2s: 0.96 3s:1.65 Col.Rate%: 1s: 0.05%; 2s:0.17% 3s:0.71 paper %. Could you help me to explain this difference of eval? thanks

fsong666 commented 3 months ago

I have solved this issue by loading the newest uniad_base_e2e.pth from github. The results are as follows: +-------------+--------+--------+--------+--------+--------+--------+ | metrics | 0.5s | 1.0s | 1.5s | 2.0s | 2.5s | 3.0s | +-------------+--------+--------+--------+--------+--------+--------+ | obj_col | 0.0000 | 0.0000 | 0.0003 | 0.0003 | 0.0007 | 0.0010 | | obj_box_col | 0.0003 | 0.0012 | 0.0013 | 0.0015 | 0.0030 | 0.0060 | | L2 | 0.3356 | 0.5075 | 0.7208 | 0.9843 | 1.2930 | 1.6530 | +-------------+--------+--------+--------+--------+--------+--------+