search

kujason / avod

Code for 3D object detection for autonomous driving
MIT License
939 stars 347 forks source link

Performance on validation set not aligned to the report in paper #10

Closed cjjsaic closed 6 years ago

cjjsaic commented 6 years ago

"car_detection_3D AP: 82.047119 67.536583 66.807381" top performance on iteration 39000, this is the top ranked by the given script, there is still a gap on the moderate, I leave the config by default to run "avod_cars_examples.config", is there anything I am missing ?

kujason commented 6 years ago

39,000 seems very early on in the training steps, did you train the network to the full 120,000 iterations? The best performing checkpoint is normally found in the later steps, usually steps 60,000 or after.

yzhou-saic commented 6 years ago

Tried "pyramid_cars_with_aug_example" on validation set, leaving all the configs by default. The best result is achieved on iteration 109000, "car_detection_3D AP: 82.985123 73.542229 67.380028", it's quite impressive, but it's still not aligned to the report in paper "car_detection_3D AP: 84.41 74.44 68.65" ??????? Thanks for the help in advance.

asharakeh commented 6 years ago

@yzhou-saic There is some variability due to random parameter initialization & Stochastic optimization procedure, a +- 1% AP is something we saw whenever we trained a new model even with the exact same configuration as an older model.

As with any other neural network, you will almost never converge to the same (local) minimum when you repeat experiments.

cjjsaic commented 6 years ago

@kujason @asharakeh Thanks for the response. Really great work and code.

ktxu1224 commented 6 years ago

@cjjsaic @asharakeh Hi， May I ask you a question met during the process of Mini-batch Generation? Traceback (most recent call last): File "scripts/preprocessing/gen_mini_batches.py", line 6, in from avod.builders.dataset_builder import DatasetBuilder File "/home/zhyan/avod/avod/builders/dataset_builder.py", line 169 new_cfg=None) -> KittiDataset: ^ SyntaxError: invalid syntax

I have placed the dataset and finished the other procedures above as ReadMe adviced, but met this problem when I tried to generate mini batch. Do you know how to solve it?

Thx a lot!

melfm commented 6 years ago

Looks like you are using Python2.7, you need to use Python3.5.

Sakura1221 commented 6 years ago

@kujason @asharakeh @yzhou-saic May I ask you a question met during the process of Mini-batch Generation? sakura@sakura-Z170X-UD3:~/avod-master$ ### python scripts/preprocessing/gen_mini_batches.py

Traceback (most recent call last): File "scripts/preprocessing/gen_mini_batches.py", line 199, in main() File "scripts/preprocessing/gen_mini_batches.py", line 120, in main car_dataset_config_path) File "/home/sakura/avod-master/avod/builders/dataset_builder.py", line 154, in load_dataset_from_config use_defaults=False) File "/home/sakura/avod-master/avod/builders/dataset_builder.py", line 191, in build_kitti_dataset return KittiDataset(cfg_copy) File "/home/sakura/avod-master/avod/datasets/kitti/kitti_dataset.py", line 85, in init .format(self.data_split, possible_splits))

ValueError: Invalid data split: trainval, possible_splits: []

I have placed the dataset correctly and finished all the procedures as ReadMe adviced, but met this problem when I run gen_mini_batches.py. Do you know how to solve it?