how to reproduce the result on kitti testset?

[zklgame]

Hi, thank you for sharing your code and trained model. I have run your trained model by test_net.sh without modifying any code about the basic config. I got some results based on 'kitti_val', and the results are:

number of files for evaluation: 3477
car_detection AP: 98.640228 88.898941 71.541313
car_orientation AP: 98.115181 87.832764 70.612312
car_detection_ground AP: 70.343857 49.159752 41.578720
car_detection_3d AP: 55.502235 36.828293 31.459877

When I was trying to change 'kitti_val' to 'kitti_test' in file test_net.py, linte 69, some error occurred:

Traceback (most recent call last):
  File "test_net.py", line 69, in <module>
    imdb, roidb, ratio_list, ratio_index = combined_roidb('kitti_test', False)
  File "/private/zhukaili/Stereo-RCNN/lib/roi_data_layer/roidb.py", line 115, in combined_roidb
    roidbs = [get_roidb(s) for s in imdb_names.split('+')]
  File "/private/zhukaili/Stereo-RCNN/lib/roi_data_layer/roidb.py", line 109, in get_roidb
    imdb = get_imdb(imdb_name)
  File "/private/zhukaili/Stereo-RCNN/lib/datasets/factory.py", line 29, in get_imdb
    return __sets[name]()
  File "/private/zhukaili/Stereo-RCNN/lib/datasets/factory.py", line 23, in <lambda>
    __sets[name] = (lambda split=split: kitti(split, data_path))
  File "/private/zhukaili/Stereo-RCNN/lib/datasets/kitti.py", line 50, in __init__
    self._image_index = self._load_image_set_index_new()
  File "/private/zhukaili/Stereo-RCNN/lib/datasets/kitti.py", line 116, in _load_image_set_index_new
    return image_index
UnboundLocalError: local variable 'image_index' referenced before assignment

It seemed that the function _load_image_set_index_new didn't do anything about the testset. So I am here for your help about how to reproduce the result on kitti testset?

Thanks.

[PeiliangLi]

Hi, thanks for your interests. If you want to evaluate on the KITTI test set, you should first use the corresponding image index (collected as a txt file like the train/val split) to build the pickle file. More details can be found in lib/datasets/kitti.py.

[judyqiu1996]

@zklgame How do you get results like this?

number of files for evaluation: 3477
car_detection AP: 98.640228 88.898941 71.541313
car_orientation AP: 98.115181 87.832764 70.612312
car_detection_ground AP: 70.343857 49.159752 41.578720
car_detection_3d AP: 55.502235 36.828293 31.459877

I only get the result pictures but not AP results.

[PeiliangLi]

@judyqiu1996 Hi, the results for each image are saved at models_stereo/result. Use the official tool (https://github.com/prclibo/kitti_eval) to evaluate the result to get APs.

[zklgame]

@zklgame How do you get results like this?

number of files for evaluation: 3477
car_detection AP: 98.640228 88.898941 71.541313
car_orientation AP: 98.115181 87.832764 70.612312
car_detection_ground AP: 70.343857 49.159752 41.578720
car_detection_3d AP: 55.502235 36.828293 31.459877

I only get the result pictures but not AP results.

1. Run test.sh and get some results.
2. Download https://github.com/prclibo/kitti_eval or other evaluation tools to evaluate the results.

[zklgame]

Hi, thanks for your interests. If you want to evaluate on the KITTI test set, you should first use the corresponding image index (collected as a txt file like the train/val split) to build the pickle file. More details can be found in lib/datasets/kitti.py.

I read the code and found that the file kitti.py and other relative files did not support for test mode. So I wonder if you have other methods to run on the test dataset? Because your public code show little about the test?