search

waymo-research / waymo-open-dataset

Waymo Open Dataset
https://www.waymo.com/open
Other
2.66k stars 608 forks source link

[3D tracking] validation and submission questions. #186

Open seoho-kang opened 4 years ago

seoho-kang commented 4 years ago

I have two question about validation and submission.

[Validation] I was going to run this command to validate my model. "bazel-bin/waymo_open_dataset/metrics/tools/compute_detection_metrics_main waymo_open_dataset/metrics/tools/prediction.bin waymo_open_dataset/metrics/tools/gt.bin"

Q0) Is it the right command to validate my model for 3D tracking? Q1) Do I use gt.bin in this directory? --> gs://waymo_open_dataset_v_1_2_0/validation/ground_truth_objects/gt.bin Q2) Which dataset should I input in my tracking main code to make prediction.bin between "validation detection dataset?" or "training detection dataset?" [Validation one] gs://waymo_open_dataset_v_1_2_0_detection_output/detection_3d/cyclist_detection_validation.bin [Training one] gs://waymo_open_dataset_v_1_2_0_detection_output/detection_3d/cyclist_detection_train.bin

[Submission] I was going to run this command to submit my model "metrics/tools/create_submission --input_filenames='/tmp/prediction.bin' --output_filename='/tmp/my_model/model' --submission_filename='metrics/tools/submission.txtpb'"

Q3) Should I input Test detection dataset to my 3D tracking main code to make prediction.bin? during submission?

peisun1115 commented 4 years ago

[Validation]

  1. Use compute_tracking_metrics for 3D tracking tasks.

  2. yes

  3. Yes, you can use those to get started. But a better 3d tracking system might need a better 3d detector which you can develop on your own.

[Submissin]

  1. you can. see my answer to 2 above also
seoho-kang commented 4 years ago

Just to confirm to make better 3d tracking system, my question is, 1) Do I get prediction.bin file through feeding validation detection dataset to my tracking code? In order to validate with gt.bin file through "bazel-bin/waymo_open_dataset/metrics/tools/compute_tracking_metrics_main waymo_open_dataset/metrics/tools/prediction.bin waymo_open_dataset/metrics/tools/gt.bin" I thought I need to get prediction.bin from validation detection dataset because gt.bin is from validation dataset.

2) For submission(test), do I get prediction.bin file through feeding test detection dataset and test ground truth dataset to my tracking code?