AssertionError: Samples in split doesn't match samples in predictions.

[hbc2020]

Dear author, Thanks for your great work. As restricted by my memory, I have to run this project on nuScenes v1.0-mini, then following error occured, the error indicates that the number of sample tokens of prediction is 162, while the number od sample tokens of annotation is only 81. Could you please give me some suggestions to solve this problem, thank you very much!

sh samples/bevformer/base/pth_evaluate.sh -d 0 Running on the GPU: 0 Loaded tensorrt plugins from /workspace/BEVFormer_tensorrt/TensorRT/lib/libtensorrt_ops.so load checkpoint from local path: checkpoints/pytorch/bevformer_r101_dcn_24ep.pth

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 162/162, 2.1 task/s, elapsed: 79s, ETA: 0s Formating bboxes of pts_bbox Start to convert detection format... [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 162/162, 20.3 task/s, elapsed: 8s, ETA: 0s Results writes to work_dirs/results_eval/pts_bbox/results_nusc.json Evaluating bboxes of pts_bbox

Loading NuScenes tables for version v1.0-mini... 23 category, 8 attribute, 4 visibility, 911 instance, 12 sensor, 120 calibrated_sensor, 31206 ego_pose, 8 log, 10 scene, 404 sample, 31206 sample_data, 18538 sample_annotation, 4 map, Done loading in 0.508 seconds.

Reverse indexing ... Done reverse indexing in 0.1 seconds.

Initializing nuScenes detection evaluation Loaded results from work_dirs/results_eval/pts_bbox/results_nusc.json. Found detections for 162 samples. Loading annotations for mini_val split from nuScenes version: v1.0-mini 100%| ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 81/81 [00:00<00:00, 327.86it/s] Loaded ground truth annotations for 81 samples. Traceback (most recent call last): File "/workspace/BEVFormer_tensorrt/tools/bevformer/evaluate_pth.py", line 138, in main() File "/workspace/BEVFormer_tensorrt/tools/bevformer/evaluate_pth.py", line 122, in main metric = dataset.evaluate(results, jsonfile_prefix='work_dirs/results_eval') #modified by bocheng.hu File "/workspace/BEVFormer_tensorrt/./third_party/bevformer/datasets/nuscenes_dataset.py", line 522, in evaluate ret_dict = self._evaluate_single(result_files[name]) File "/workspace/BEVFormer_tensorrt/./third_party/bevformer/datasets/nuscenes_dataset.py", line 869, in _evaluate_single self.nusc_eval = NuScenesEval_custom( File "/workspace/BEVFormer_tensorrt/./third_party/bevformer/datasets/nuscenes_eval.py", line 672, in init assert set(self.pred_boxes.sample_tokens) == set( AssertionError: Samples in split doesn't match samples in predictions.

[DerryHub]

This project doesn't support nuScenes v1.0-mini yet. Maybe you can modify the BEVFormer_tensorrt/third_party/bevformer/datasets/nuscenes_eval.py file to support it.

[hbc2020]

Thanks for your reply, I will try to modify the code to make it support nuScenes v1.0-mini.

[sainttelant]

@hbc2020 ， hello， i also tried to inference the nuscences mini-data, and encountered the same issue with you, so how did you modify the code? please give me some suggestions, thanks a lot

[XdanMT]

@sainttelant Hi, the root cause of this error is that during the dataset preparation phase, the mini dataset is divided into train and val instead of mini_train and mini_val.

If you want to use nuscenes-mini to evaluate the result, you can simply modify the code in line 78 of file “tools/bevformer/create_data.py” as follows: train_scenes = splits.mini_train val_scenes = splits.mini_val

Then, prepare the mini dataset again: python tools/bevformer/create_data.py nuscenes --root-path ./data/nuscenes-mini --out-dir ./data/nuscenes-mini --extra-tag nuscenes --version v1.0 --canbus ./data

By the way, you can output the split of the mini dataset. The correct set is:

train_scenes: ['scene-0061', 'scene-0553', 'scene-0655', 'scene-0757', 'scene-0796', 'scene-1077', 'scene-1094', 'scene-1100'] val_scenes: ['scene-0103', 'scene-0916']

[LiaoChengkk]

@sainttelant Hi, the root cause of this error is that during the dataset preparation phase, the mini dataset is divided into train and val instead of mini_train and mini_val.

If you want to use nuscenes-mini to evaluate the result, you can simply modify the code in line 78 of file “tools/bevformer/create_data.py” as follows: train_scenes = splits.mini_train val_scenes = splits.mini_val

Then, prepare the mini dataset again: python tools/bevformer/create_data.py nuscenes --root-path ./data/nuscenes-mini --out-dir ./data/nuscenes-mini --extra-tag nuscenes --version v1.0 --canbus ./data

By the way, you can output the split of the mini dataset. The correct set is:

train_scenes: ['scene-0061', 'scene-0553', 'scene-0655', 'scene-0757', 'scene-0796', 'scene-1077', 'scene-1094', 'scene-1100'] val_scenes: ['scene-0103', 'scene-0916']

@XdanMT Thanks! You help me a lot. Btw, in line 76 should be changed to available_vers = ["v1.0-trainval","v1.0-mini"] and in line 765 should be changed to train_version = f"{args.version}-mini"