search

Nightmare-n / GD-MAE

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds (CVPR 2023)
Apache License 2.0
114 stars 6 forks source link

Performance on Waymo dataset lower than reported #17

Open codename1995 opened 1 year ago

codename1995 commented 1 year ago

Hi,

thank you for your great work. It's highly inspired.

I do have a few questions though. I attempted to reproduce results on Waymo Open Datasets but the performance is a bit lower than reported in the paper. I am wondering is it caused by the adaption I made or I missed anything.

What I did are

  1. Follow the preprocessing instruction to process WOD.
  2. Use 20% data to do pre-training (SSL) and training (supervised learning) by setting SAMPLED_INTERVAL['train'] to 5 in both gd_mae_ssl.yaml and gd_mae.yaml.
  3. Set FILTER_EMPTY_BOXES_FOR_TRAIN to False for both pre-training and training in waymo_dataset.yaml. Otherwise I run into RecursionError: maximum recursion depth exceeded in comparison issue because GT boxes containing are always empty.
  4. Batch size. One major difference is that, due to the GPU memory limit, I have to set batch_size*GPUs to 4x8 and 2x8 for pre-training and training, respectively, instead of 8x4 in the paper.

What I obtained are

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.659692] [mAPH 0.655148]
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.705515] [mAPH 0.627452]
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.68869] [mAPH 0.676339]

which is 1~2 points lower than reported in Table 4. Do you think it is caused by the reduction of batch size or it's a normal variation?

Later, I realized that the FILTER_EMPTY_BOXES_FOR_TRAIN (in step 3) can be set as True as in the original code for (supervised) training. But the performance goes very low, like 0.06/0.06 for vehicles, 0.10/0.09 for pedestrian. I am wondering what should this variable be for pre-training and training?

Thanks in advance.

jisoo0-0 commented 5 months ago

Hi, I have some question on your work.

How can you get the waymo dataset's result? Because i just can get below result when i run the code for waymo dataset.

2024-04-15 12:24:41,593 INFO Average predicted number of objects(39987 samples): 100.000 2024-04-15 12:24:44.026785: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 39987it [00:40, 990.05it/s] results saved to /home/workspace/jisoocv/GD-MAE/output/kitti_models/graph_rcnn_po/default/eval/epoch_no_number/val/default/final_result/data/detection_pred.bin 2024-04-15 12:25:39,637 INFO
2024-04-15 12:25:41,141 INFO Result is save to /home/workspace/jisoocv/GD-MAE/output/kitti_models/graph_rcnn_po/default/eval/epoch_no_number/val/default 2024-04-15 12:25:41,141 INFO ****Evaluation done.*****