search

Sense-GVT / Fast-BEV

Fast-BEV: A Fast and Strong Bird’s-Eye View Perception Baseline
Other
596 stars 91 forks source link

Why i try run tools/eval.py mAP==0.0 NDS==0.0 #23

Closed zouzouwei closed 1 year ago

zouzouwei commented 1 year ago

image The results I got mAP=0.000 NDS=0.000. The following is the step I run code

1, dataset nuscenes version v1.0-mini

Modify the code： tools/create_data.py Line 188-212

parser.add_argument('dataset', metavar='nuscenes', help='name of the dataset')
parser.add_argument(
    '--root-path',
    type=str,
    default='./data/nuscenes',
    help='specify the root path of dataset')
parser.add_argument(
    '--version',
    type=str,
    default='v1.0-mini',
    required=False,
    help='specify the dataset version, no need for kitti')
parser.add_argument(
    '--max-sweeps',
    type=int,
    default=10,
    required=False,
    help='specify sweeps of lidar per example')
parser.add_argument(
    '--out-dir',
    type=str,
    default='./data/nuscenes',
    required=False,
    help='name of info pkl')
parser.add_argument('--extra-tag', type=str, default='nuscenes')

tools/data_converter/nuscenes_converter.py Line 45

 available_vers = ['v1.0-trainval', 'v1.0-test', 'v1.0-mini']

 elif version == 'v1.0-mini':
        train_scenes = splits.mini_train
        val_scenes = splits.mini_val

tools/data_converter/nuscenes_seq_converter.py Line 14-16

    for set in ['val', 'train', ]:
        # if set in ['val', 'train']:
        #     continue

later, i run python tools/create_data.py nuscenes python tools/data_converter/nuscenes_seq_converter.py ps: cp nuscense_trainval_dataset map/ nuscense_v1.0-mini /map/ Got these: image

2.run test.py

python tools/eval.py configs/fastbev/exp/paper/fastbev_m0_r18_s256x704_v200x200x4_c192_d2_f4.py --out temp_out/temp.pkl --eval bbox later:

======
Loading NuScenes tables for version v1.0-mini...
23 category,
8 attribute,
4 visibility,
911 instance,
12 sensor,
120 calibrated_sensor,
31206 ego_pose,
8 log,
10 scene,
404 sample,
31206 sample_data,
18538 sample_annotation,
4 map,
Done loading in 0.376 seconds.
======
Reverse indexing ...
Done reverse indexing in 0.1 seconds.
======
lane thickness: 2
lane thickness: 2
lane thickness: 2
lane thickness: 2
lane thickness: 2

loading results from temp_out/temp.pkl
Start to convert detection format...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 18.6 task/s, elapsed: 4s, ETA:     0s
Results writes to /tmp/tmpow_oc5yk/results/results_nusc.json
mAP: 0.0000                                                                                                                                                                                         
mATE: 1.0000
mASE: 1.0000
mAOE: 1.0000
mAVE: 1.0000
mAAE: 1.0000
NDS: 0.0000
Eval time: 1.0s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.000   1.000   1.000   1.000   1.000   1.000
truck   0.000   1.000   1.000   1.000   1.000   1.000
bus     0.000   1.000   1.000   1.000   1.000   1.000
trailer 0.000   1.000   1.000   1.000   1.000   1.000
construction_vehicle    0.000   1.000   1.000   1.000   1.000   1.000
pedestrian      0.000   1.000   1.000   1.000   1.000   1.000
motorcycle      0.000   1.000   1.000   1.000   1.000   1.000
bicycle 0.000   1.000   1.000   1.000   1.000   1.000
traffic_cone    0.000   1.000   1.000   nan     nan     nan
barrier 0.000   1.000   1.000   1.000   nan     nan
{'pts_bbox_NuScenes/car_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/car_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/car_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/car_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/car_trans_err': 1.0, 'pts_bbox_NuScenes/car_scale_err': 1.0, 'pts_bbox_NuScenes/car_orient_err': 1.0, 'pts_bbox_NuScenes/car_vel_err': 1.0, 'pts_bbox_NuScenes/car_attr_err': 1.0, 'pts_bbox_NuScenes/mATE': 1.0, 'pts_bbox_NuScenes/mASE': 1.0, 'pts_bbox_NuScenes/mAOE': 1.0, 'pts_bbox_NuScenes/mAVE': 1.0, 'pts_bbox_NuScenes/mAAE': 1.0, 'pts_bbox_NuScenes/truck_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/truck_trans_err': 1.0, 'pts_bbox_NuScenes/truck_scale_err': 1.0, 'pts_bbox_NuScenes/truck_orient_err': 1.0, 'pts_bbox_NuScenes/truck_vel_err': 1.0, 'pts_bbox_NuScenes/truck_attr_err': 1.0, 'pts_bbox_NuScenes/trailer_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/trailer_trans_err': 1.0, 'pts_bbox_NuScenes/trailer_scale_err': 1.0, 'pts_bbox_NuScenes/trailer_orient_err': 1.0, 'pts_bbox_NuScenes/trailer_vel_err': 1.0, 'pts_bbox_NuScenes/trailer_attr_err': 1.0, 'pts_bbox_NuScenes/bus_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/bus_trans_err': 1.0, 'pts_bbox_NuScenes/bus_scale_err': 1.0, 'pts_bbox_NuScenes/bus_orient_err': 1.0, 'pts_bbox_NuScenes/bus_vel_err': 1.0, 'pts_bbox_NuScenes/bus_attr_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_trans_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_scale_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_orient_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_vel_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_attr_err': 1.0, 'pts_bbox_NuScenes/bicycle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bicycle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bicycle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/bicycle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/bicycle_trans_err': 1.0, 'pts_bbox_NuScenes/bicycle_scale_err': 1.0, 'pts_bbox_NuScenes/bicycle_orient_err': 1.0, 'pts_bbox_NuScenes/bicycle_vel_err': 1.0, 'pts_bbox_NuScenes/bicycle_attr_err': 1.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/motorcycle_trans_err': 1.0, 'pts_bbox_NuScenes/motorcycle_scale_err': 1.0, 'pts_bbox_NuScenes/motorcycle_orient_err': 1.0, 'pts_bbox_NuScenes/motorcycle_vel_err': 1.0, 'pts_bbox_NuScenes/motorcycle_attr_err': 1.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/pedestrian_trans_err': 1.0, 'pts_bbox_NuScenes/pedestrian_scale_err': 1.0, 'pts_bbox_NuScenes/pedestrian_orient_err': 1.0, 'pts_bbox_NuScenes/pedestrian_vel_err': 1.0, 'pts_bbox_NuScenes/pedestrian_attr_err': 1.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/traffic_cone_trans_err': 1.0, 'pts_bbox_NuScenes/traffic_cone_scale_err': 1.0, 'pts_bbox_NuScenes/traffic_cone_orient_err': nan, 'pts_bbox_NuScenes/traffic_cone_vel_err': nan, 'pts_bbox_NuScenes/traffic_cone_attr_err': nan, 'pts_bbox_NuScenes/barrier_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/barrier_trans_err': 1.0, 'pts_bbox_NuScenes/barrier_scale_err': 1.0, 'pts_bbox_NuScenes/barrier_orient_err': 1.0, 'pts_bbox_NuScenes/barrier_vel_err': nan, 'pts_bbox_NuScenes/barrier_attr_err': nan, 'pts_bbox_NuScenes/NDS': 0.0, 'pts_bbox_NuScenes/mAP': 0.0}
{'pts_bbox_NuScenes/car_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/car_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/car_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/car_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/car_trans_err': 1.0, 'pts_bbox_NuScenes/car_scale_err': 1.0, 'pts_bbox_NuScenes/car_orient_err': 1.0, 'pts_bbox_NuScenes/car_vel_err': 1.0, 'pts_bbox_NuScenes/car_attr_err': 1.0, 'pts_bbox_NuScenes/mATE': 1.0, 'pts_bbox_NuScenes/mASE': 1.0, 'pts_bbox_NuScenes/mAOE': 1.0, 'pts_bbox_NuScenes/mAVE': 1.0, 'pts_bbox_NuScenes/mAAE': 1.0, 'pts_bbox_NuScenes/truck_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/truck_trans_err': 1.0, 'pts_bbox_NuScenes/truck_scale_err': 1.0, 'pts_bbox_NuScenes/truck_orient_err': 1.0, 'pts_bbox_NuScenes/truck_vel_err': 1.0, 'pts_bbox_NuScenes/truck_attr_err': 1.0, 'pts_bbox_NuScenes/trailer_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/trailer_trans_err': 1.0, 'pts_bbox_NuScenes/trailer_scale_err': 1.0, 'pts_bbox_NuScenes/trailer_orient_err': 1.0, 'pts_bbox_NuScenes/trailer_vel_err': 1.0, 'pts_bbox_NuScenes/trailer_attr_err': 1.0, 'pts_bbox_NuScenes/bus_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/bus_trans_err': 1.0, 'pts_bbox_NuScenes/bus_scale_err': 1.0, 'pts_bbox_NuScenes/bus_orient_err': 1.0, 'pts_bbox_NuScenes/bus_vel_err': 1.0, 'pts_bbox_NuScenes/bus_attr_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_trans_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_scale_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_orient_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_vel_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_attr_err': 1.0, 'pts_bbox_NuScenes/bicycle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bicycle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bicycle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/bicycle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/bicycle_trans_err': 1.0, 'pts_bbox_NuScenes/bicycle_scale_err': 1.0, 'pts_bbox_NuScenes/bicycle_orient_err': 1.0, 'pts_bbox_NuScenes/bicycle_vel_err': 1.0, 'pts_bbox_NuScenes/bicycle_attr_err': 1.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/motorcycle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/motorcycle_trans_err': 1.0, 'pts_bbox_NuScenes/motorcycle_scale_err': 1.0, 'pts_bbox_NuScenes/motorcycle_orient_err': 1.0, 'pts_bbox_NuScenes/motorcycle_vel_err': 1.0, 'pts_bbox_NuScenes/motorcycle_attr_err': 1.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/pedestrian_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/pedestrian_trans_err': 1.0, 'pts_bbox_NuScenes/pedestrian_scale_err': 1.0, 'pts_bbox_NuScenes/pedestrian_orient_err': 1.0, 'pts_bbox_NuScenes/pedestrian_vel_err': 1.0, 'pts_bbox_NuScenes/pedestrian_attr_err': 1.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/traffic_cone_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/traffic_cone_trans_err': 1.0, 'pts_bbox_NuScenes/traffic_cone_scale_err': 1.0, 'pts_bbox_NuScenes/traffic_cone_orient_err': nan, 'pts_bbox_NuScenes/traffic_cone_vel_err': nan, 'pts_bbox_NuScenes/traffic_cone_attr_err': nan, 'pts_bbox_NuScenes/barrier_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/barrier_trans_err': 1.0, 'pts_bbox_NuScenes/barrier_scale_err': 1.0, 'pts_bbox_NuScenes/barrier_orient_err': 1.0, 'pts_bbox_NuScenes/barrier_vel_err': nan, 'pts_bbox_NuScenes/barrier_attr_err': nan, 'pts_bbox_NuScenes/NDS': 0.0, 'pts_bbox_NuScenes/mAP': 0.0}

Why did it turn out like this?

ps: i try run code: python tools/test.py configs/fastbev/exp/paper/fastbev_m0_r18_s256x704_v200x200x4_c192_d2_f4.py pretrained_models/cascade_mask_rcnn_r18_fpn_coco-mstrain_3x_20e_nuim_bbox_mAP_0.5110_segm_mAP_0.4070.pth --eval bbox

result same

zouzouwei commented 1 year ago

image This is the environment where I run.

Ignite616 commented 1 year ago

I also used nuScenes-mini datasets, and fellow you step. but when i run eval.py, i have an error. like this AssertionError: NuScenesMultiView_Map_Dataset2: Database version not found: ./data/nuscenes/v1.0-trainval it seems that dataset = build_from_cfg(cfg, DATASETS, default_args), not deal with mini. how do you do work? thank you!

Ignite616 commented 1 year ago

I find you use cp nuscense_trainval_dataset map/ nuscense_v1.0-mini /map/ whar means，copy nuscense_trainval_dataset to map/ nuscense_v1.0-mini /map/? why do this?

zouzouwei commented 1 year ago

I find you use cp nuscense_trainval_dataset map/ nuscense_v1.0-mini /map/ whar means，copy nuscense_trainval_dataset to map/ nuscense_v1.0-mini /map/? why do this?

because , model will lodaing map/expansion information。

zouzouwei commented 1 year ago

I also used nuScenes-mini datasets, and fellow you step. but when i run eval.py, i have an error. like this AssertionError: NuScenesMultiView_Map_Dataset2: Database version not found: ./data/nuscenes/v1.0-trainval it seems that dataset = build_from_cfg(cfg, DATASETS, default_args), not deal with mini. how do you do work? thank you!

sorry, i forget write this error： mmdet3d/datasets/nuscenes_monocular_dataset_map_2.py L7

        self.nusc = NuScenes(version='v1.0-mini', dataroot=self.data_root, verbose=True)
Ignite616 commented 1 year ago

I find you use cp nuscense_trainval_dataset map/ nuscense_v1.0-mini /map/ whar means，copy nuscense_trainval_dataset to map/ nuscense_v1.0-mini /map/? why do this?

because , model will lodaing map/expansion information。

I have an error, like this: No such file or directory: './data/nuscenes/maps/expansion/singapore-hollandvillage.json' Where can I find singapore-hollandvillage.json or how to make it? can I see you catalogue, thanks!

zouzouwei commented 1 year ago

I find you use cp nuscense_trainval_dataset map/ nuscense_v1.0-mini /map/ whar means，copy nuscense_trainval_dataset to map/ nuscense_v1.0-mini /map/? why do this?

because , model will lodaing map/expansion information。

I have an error, like this: No such file or directory: './data/nuscenes/maps/expansion/singapore-hollandvillage.json' Where can I find singapore-hollandvillage.json or how to make it? can I see you catalogue, thanks!

you open this url : https://www.nuscenes.org/nuscenes#download sing up account

find here

image

Ignite616 commented 1 year ago

Thank you for your reply. I ran tets.py and got the test results. You can try this, I think eval.py no input model. python tools/test.py configs/fastbev/exp/paper/fastbev_m1_r18_s320x880_v200x200x4_c192_d2_f4.py download_model/epoch_20.pth --eval bbox 2023-02-21 19-49-58 的屏幕截图

zouzouwei commented 1 year ago

Thank you for your reply. I ran tets.py and got the test results. You can try this, I think eval.py no input model. python tools/test.py configs/fastbev/exp/paper/fastbev_m1_r18_s320x880_v200x200x4_c192_d2_f4.py download_model/epoch_20.pth --eval bbox 2023-02-21 19-49-58 的屏幕截图

Thank you for your prompt . I solve this quesation

Ignite616 commented 1 year ago

当您运行Fast-Bev/tools/test.py时，使用M1模型测试在一帧的六张图片需要多长时间？我在T1200（相当于RTX 1080的算力）上进行了测试，测试一帧需要2秒左右。它与论文中的表1和表11的耗时有很大不同。我使用M4模型测试时，每帧也用了4秒左右。 你测试了多久？如果你用其他模型测试它，你也可以详细说明下。 当我训练时，我在两张TITAN RTX TU102(相当于RTX 3080的算力)上训练m1模型，每张卡的批大小为8，使用的nuScenes数据集，训练了21个小时，第一个周期只训练到4530/8007，这训练速度也太慢了吧？ 我在T1200（相当于RTX 1080的算力）上训练m1模型，批大小为1，使用的nuScenes-mini数据集，训练了5个小时，只训练到第三个周期的730/1640。训练集只有大约320张照片，为什么这里是1640？