search

waymo-research / waymo-open-dataset

Waymo Open Dataset
https://www.waymo.com/open
Other
2.72k stars 616 forks source link

Number of points in 3d-BBox #156

Open stalkermustang opened 4 years ago

stalkermustang commented 4 years ago

Hello. In my research and debugging i found that number of points field in labeled data is calculated using only BEV-methods, without z-dimension and height. Is that right? And how this affect on WAYMO-leaderboard in 3d detection? Potentially ground truth bbox may has 0 points, but some points located near it (i.e.bottom of car, on road) - above or under. Also this affects to level-1 and level-2 samples distribution.

peisun1115 commented 4 years ago

The number of points in box is computed with z-dim. There can be some differences if you recompute. The difference is ok. Feel free to use whatever you compute in your model development. Whereas we will use values pre-populated in the dataset for metrics computation such that everyone has the same standard.

tianweiy commented 4 years ago

@peisun1115 Can you direct me to some official code to compute the num_points_in_box? I tried to use https://github.com/waymo-research/waymo-open-dataset/blob/c6927afe3e78c876d40ff2df7eb7fbc6007b5505/waymo_open_dataset/utils/box_utils.py#L73

I compare the results with the original num_point in the data, and the mismatch is really large.(I define the mismatch to be that one method filters out the box while the other doesn't)

Here are some statistics, among 2400k boxes in the validation set.

match: 248317 use original num_point to filter: +3634 use the util file to filter : +85052

It seems that we won't have a good way to filter out zero point prediction boxes without knowing how the gt num_point is computed ..

peisun1115 commented 4 years ago

@tianweiy why do you need to compute that on your own? to compute metrics, you only need to filter based on what we provided. For other purposes, use any lib to compute.

tianweiy commented 4 years ago

Maybe I misunderstood something. But for our submission, do we need to filter those zero lidar points prediction? https://github.com/waymo-research/waymo-open-dataset/issues/151#issuecomment-622899231 here seems to indicate that it improves 10 percent after filtering?

peisun1115 commented 4 years ago

no. you do not need to filter boxes with 0 points in your predictions. You are free to give any boxes. But for ground truth, you need to filter. That filtering needs to be based on the number of points we provide.

tianweiy commented 4 years ago

I see. thanks for the clarification.

tianweiy commented 4 years ago

@peisun1115 I have one further question. Do you filter out zero point gt boxes during the training for StarNet / baseline in the paper?

peisun1115 commented 4 years ago

yes, we do.