Waymo evaluation issue

[mincheoree]

Hello, thanks for great works! During Waymo dataset evaluation, I encountered 0 mAP and GT objects.

I searched in PCDet issue, the author says that Waymo 1.0.0 is incompatible with evaluation, and they use Waymo 1.2.0 dataset as default, referring to this: https://github.com/open-mmlab/OpenPCDet/issues/1102

However, in your AD-PT paper, you mentioned that you used Waymo 1.0.0 as your experiment setting. As your codebase is based on PCDet, may I ask if you also encountered this 0 AP error during waymo evaluation? Is Waymo 1.0.0 okay for evaluation?

[BOBrown]

@mincheoree hi, here is my conda envs of waymo: # Name | Version waymo-open-dataset-tf-2-0-0 | 1.3.1

please consider using

pip3 install waymo-open-dataset-tf-2-0-0

---

plus, I found in your log that the parsing ground truth is [0,7] [0]. No GT is loaded, and it is recommended that you check the loaded waymo gt_infos_train.pkl

[mincheoree]

@BOBrown Thank you for providing specific environment version! I will try it out. For your other suggestion, I cannot find gt_infos pickle. In your DB readme, the waymo dataset is processed as below, and I could not find gt_infos_train.pkl file.

[BOBrown]

@mincheoree waymo_processed_data_v0_5_0_infos_train.pkl and waymo_processed_data_v0_5_0_infos_val.pkl

[mincheoree]

@BOBrown After moving to waymo dataset version 1.2, the issue is solved.

[mincheoree]

@BOBrown @sky-fly97 I got another question, I trained SECOND model (without IOU branch) with 3% waymo dataset from scratch (without pretrained model) with provided yaml, waymo 3% dataset config, but got quite different result with your report in README. Your result is as following:		Data amount	Overall	Vehicle	Pedestrian	Cyclist
[SECOND (From scratch)]()	3%	52.00 / 37.70	58.11 / 57.44	51.34 / 27.38	46.57 / 28.28

While I get

	Data amount	Overall	Vehicle	Pedestrian	Cyclist
[SECOND (From scratch)]()	3%	48.94 /41.96	55.74/ 54.96	48.25 / 32.78	42.84 / 38.14

Was your test sample interval set to 5? This is quite different performance.

[sky-fly97]

@BOBrown @sky-fly97 I got another question, I trained SECOND model (without IOU branch) with 3% waymo dataset from scratch (without pretrained model) with provided yaml, waymo 3% dataset config, but got quite different result with your report in README. Your result is as following:

Data amount Overall Vehicle Pedestrian Cyclist SECOND (From scratch) 3% 52.00 / 37.70 58.11 / 57.44 51.34 / 27.38 46.57 / 28.28 While I get

Data amount Overall Vehicle Pedestrian Cyclist SECOND (From scratch) 3% 48.94 /41.96 55.74/ 54.96 48.25 / 32.78 42.84 / 38.14 Was your test sample interval set to 5? This is quite different performance.

Hello，did you use VoxelBackBone8x as 3d backbone while using second? Note that we use VoxelResBackBone8x as the 3D backbone. Please change it at here.