Problem with Checkpoint

[alfredgu001324]

work_dir: ./work_dirs/vectormapnet collecting samples... collected 81 samples in 0.00s 2023-07-05 14:38:13,830 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2023-07-05 14:38:13,830 - mmcv - INFO - Use load_from_openmmlab loader 2023-07-05 14:38:13,878 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

missing keys in source state_dict: layer3.0.conv2.conv_offset.weight, layer3.0.conv2.conv_offset.bias, layer3.1.conv2.conv_offset.weight, layer3.1.conv2.conv_offset.bias, layer3.2.conv2.conv_offset.weight, layer3.2.conv2.conv_offset.bias, layer3.3.conv2.conv_offset.weight, layer3.3.conv2.conv_offset.bias, layer3.4.conv2.conv_offset.weight, layer3.4.conv2.conv_offset.bias, layer3.5.conv2.conv_offset.weight, layer3.5.conv2.conv_offset.bias, layer4.0.conv2.conv_offset.weight, layer4.0.conv2.conv_offset.bias, layer4.1.conv2.conv_offset.weight, layer4.1.conv2.conv_offset.bias, layer4.2.conv2.conv_offset.weight, layer4.2.conv2.conv_offset.bias

Use load_from_local loader The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1x1.weight

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 7.6 task/s, elapsed: 11s, ETA: 0sstart evaluation! len of the results 81 [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 11455.6 task/s, elapsed: 0s, ETA: 0s Done! ----------thershold:0.5---------- results path: ./work_dirs/vectormapnet/results_nuscence.pkl metric: chamfer threshold: -0.5 update: True fix_interval: False class_num: ['ped_crossing', 'divider', 'contours'] Formatting ... Data formatting done in 1.658856s!! cls:ped_crossing done in 0.008414s!! cls:divider done in 0.015380s!! cls:contours done in 0.020888s!!

+--------------+-----+------+--------+-------+ | class | gts | dets | recall | ap | +--------------+-----+------+--------+-------+ | ped_crossing | 76 | 656 | 0.605 | 0.435 | | divider | 460 | 1149 | 0.617 | 0.425 | | contours | 282 | 1030 | 0.245 | 0.093 | +--------------+-----+------+--------+-------+ | mAP | | | | 0.318 | +--------------+-----+------+--------+-------+ ----------thershold:1---------- results path: ./work_dirs/vectormapnet/results_nuscence.pkl metric: chamfer threshold: -1 update: False fix_interval: False class_num: ['ped_crossing', 'divider', 'contours'] Formatting ... Data formatting done in 1.180227s!! cls:ped_crossing done in 0.011127s!! cls:divider done in 0.016809s!! cls:contours done in 0.020043s!!

+--------------+-----+------+--------+-------+ | class | gts | dets | recall | ap | +--------------+-----+------+--------+-------+ | ped_crossing | 76 | 656 | 0.934 | 0.887 | | divider | 460 | 1149 | 0.876 | 0.806 | | contours | 282 | 1030 | 0.628 | 0.459 | +--------------+-----+------+--------+-------+ | mAP | | | | 0.717 | +--------------+-----+------+--------+-------+ ----------thershold:1.5---------- results path: ./work_dirs/vectormapnet/results_nuscence.pkl metric: chamfer threshold: -1.5 update: False fix_interval: False class_num: ['ped_crossing', 'divider', 'contours'] Formatting ... Data formatting done in 1.196037s!! cls:ped_crossing done in 0.013920s!! cls:divider done in 0.015027s!! cls:contours done in 0.022493s!!

+--------------+-----+------+--------+-------+ | class | gts | dets | recall | ap | +--------------+-----+------+--------+-------+ | ped_crossing | 76 | 656 | 0.974 | 0.956 | | divider | 460 | 1149 | 0.946 | 0.892 | | contours | 282 | 1030 | 0.794 | 0.669 | +--------------+-----+------+--------+-------+ | mAP | | | | 0.839 | +--------------+-----+------+--------+-------+ ped_crossing: 0.7592802941799164 divider: 0.7075984378655752 contours: 0.40709205220143 map: 0.6246569280823072 VectormapNet Evaluation Results: {'mAP': 0.6246569280823072} {'mAP': 0.6246569280823072}

[alfredgu001324]

As you can see, the result is quite poor with the uploaded checkpoint on the nuscenes mini. I am wondering why this is the case?

[colahe]

As you can see, the result is quite poor with the uploaded checkpoint on the nuscenes mini. I am wondering why this is the case?

work_dir: ./work_dirs/vectormapnet collecting samples... collected 81 samples in 0.00s 2023-07-05 14:38:13,830 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2023-07-05 14:38:13,830 - mmcv - INFO - Use load_from_openmmlab loader 2023-07-05 14:38:13,878 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

missing keys in source state_dict: layer3.0.conv2.conv_offset.weight, layer3.0.conv2.conv_offset.bias, layer3.1.conv2.conv_offset.weight, layer3.1.conv2.conv_offset.bias, layer3.2.conv2.conv_offset.weight, layer3.2.conv2.conv_offset.bias, layer3.3.conv2.conv_offset.weight, layer3.3.conv2.conv_offset.bias, layer3.4.conv2.conv_offset.weight, layer3.4.conv2.conv_offset.bias, layer3.5.conv2.conv_offset.weight, layer3.5.conv2.conv_offset.bias, layer4.0.conv2.conv_offset.weight, layer4.0.conv2.conv_offset.bias, layer4.1.conv2.conv_offset.weight, layer4.1.conv2.conv_offset.bias, layer4.2.conv2.conv_offset.weight, layer4.2.conv2.conv_offset.bias

Use load_from_local loader The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1x1.weight

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 7.6 task/s, elapsed: 11s, ETA: 0sstart evaluation! len of the results 81 [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 11455.6 task/s, elapsed: 0s, ETA: 0s Done! ---------_-thershold:0.5----------_ results path: ./work_dirs/vectormapnet/results_nuscence.pkl metric: chamfer threshold: -0.5 update: True fix_interval: False class_num: ['ped_crossing', 'divider', 'contours'] Formatting ... Data formatting done in 1.658856s!! cls:ped_crossing done in 0.008414s!! cls:divider done in 0.015380s!! cls:contours done in 0.020888s!!

+--------------+-----+------+--------+-------+ | class | gts | dets | recall | ap | +--------------+-----+------+--------+-------+ | pedcrossing | 76 | 656 | 0.605 | 0.435 | | divider | 460 | 1149 | 0.617 | 0.425 | | contours | 282 | 1030 | 0.245 | 0.093 | +--------------+-----+------+--------+-------+ | mAP | | | | 0.318 | +--------------+-----+------+--------+-------+ ----------thershold:1----------_ results path: ./work_dirs/vectormapnet/results_nuscence.pkl metric: chamfer threshold: -1 update: False fix_interval: False class_num: ['ped_crossing', 'divider', 'contours'] Formatting ... Data formatting done in 1.180227s!! cls:ped_crossing done in 0.011127s!! cls:divider done in 0.016809s!! cls:contours done in 0.020043s!!

+--------------+-----+------+--------+-------+ | class | gts | dets | recall | ap | +--------------+-----+------+--------+-------+ | pedcrossing | 76 | 656 | 0.934 | 0.887 | | divider | 460 | 1149 | 0.876 | 0.806 | | contours | 282 | 1030 | 0.628 | 0.459 | +--------------+-----+------+--------+-------+ | mAP | | | | 0.717 | +--------------+-----+------+--------+-------+ ----------thershold:1.5----------_ results path: ./work_dirs/vectormapnet/results_nuscence.pkl metric: chamfer threshold: -1.5 update: False fix_interval: False class_num: ['ped_crossing', 'divider', 'contours'] Formatting ... Data formatting done in 1.196037s!! cls:ped_crossing done in 0.013920s!! cls:divider done in 0.015027s!! cls:contours done in 0.022493s!!

+--------------+-----+------+--------+-------+ | class | gts | dets | recall | ap | +--------------+-----+------+--------+-------+ | ped_crossing | 76 | 656 | 0.974 | 0.956 | | divider | 460 | 1149 | 0.946 | 0.892 | | contours | 282 | 1030 | 0.794 | 0.669 | +--------------+-----+------+--------+-------+ | mAP | | | | 0.839 | +--------------+-----+------+--------+-------+ ped_crossing: 0.7592802941799164 divider: 0.7075984378655752 contours: 0.40709205220143 map: 0.6246569280823072 VectormapNet Evaluation Results: {'mAP': 0.6246569280823072} {'mAP': 0.6246569280823072}

Hi, may I ask where you did the test? colab?

[alfredgu001324]

@colahe On my local machine

[colahe]

@colahe On my local machine Can you give me a peek at the instructions for the tests entered? I type: python tools/test.py configs/vectormapnet.py ./checkpoint/vectormapnet.pth --format-only but it always appears: CUDA out of memory. Tried to allocate 1.05 GiB (GPU 0; 5.80 GiB total capacity; 2.91 GiB already allocated; 686.94 MiB free; 3.72 GiB reserved in total by PyTorch), thanks a lot!

[alfredgu001324]

@colahe Sorry for the late reply. It's been a while since I explored this repo. I don't quite remember the specific arguments but I think it is a similar command (maybe the same command that they posted)

[colahe]

@colahe Sorry for the late reply. It's been a while since I explored this repo. I don't quite remember the specific arguments but I think it is a similar command (maybe the same command that they posted)

Thanks for your reply, i have solved it!!!

[XYyao16]

Hello, I also faced this situation of very poor evaluation performance. It was the same as yours. How did you solve it? Is there any problem that could not be done properly?

[alfredgu001324]

I do not quite remember, did you try on the entire val set? You can try visualizing it to see whether it is really poor or not