AP is very lower

[alsun-oven]

I use your model to eval. But the map is very low. only 72.12 in hard

[pangsu0613]

This is the evaluation result that I got: Predict_test: False sparse_shape: [ 41 1600 1408] num_class is : 1 load existing model Restoring parameters from ../model_dir/voxelnet-30950.tckpt load existing model for fusion layer Restoring parameters from /home/xi/CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt remain number of infos: 3769 Generate output labels... [100.0%][===================>][12.63it/s][04:52>00:00] generate label finished(12.86/s). start eval: validation_loss: 0.14488257023894788 avg example to torch time: 1.241 ms avg prep time: 10.736 ms Car AP@0.70, 0.70, 0.70: bbox AP:99.33, 93.75, 92.89 bev AP:96.51, 92.37, 89.41 3d AP:92.74, 82.90, 77.75 aos AP:99.17, 93.29, 92.11 Car AP@0.70, 0.50, 0.50: bbox AP:99.33, 93.75, 92.89 bev AP:99.38, 96.27, 95.56 3d AP:99.55, 96.19, 95.34 aos AP:99.17, 93.29, 92.11

Did you configure everything correctly based on the readme?

[yinggo]

@pangsu0613 I train with the code, but the eval result is very low. Plz help to check my log.txt log.txt, How can I train to get the high accuracy model? Should I change some parameters in config?

[pangsu0613]

Hello @yinggo, I had a look at your log.txt, the numbers you got is quite weird, they are very close to 0.0. I guess maybe there is something wrong with the KITTI info pickle files or the evaluation script. Do you have original SECOND installed? If yes, you need to remove original SECOND directory from $PYTHONPATH and add CLOCs directory, or it may call some original SECOND functions inappropriately. Also, can you get good performance using my pretrained model?

[yinggo]

@pangsu0613 thanks for your advice. I did not install second. I evaluate the pretrained model, it also get low accuracy:

Predict_test:  False
sparse_shape: [  41 1600 1408]
num_class is : 1
load existing model
load existing model for fusion layer
Restoring parameters from ../CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt
remain number of infos: 3769
Generate output labels...
[100.0%][===================>][10.41it/s][05:56>00:00]     
generate label finished(10.57/s). start eval:
validation_loss: 325.96742813958514
avg example to torch time: 1.646 ms
avg prep time: 8.815 ms
Car AP@0.70, 0.70, 0.70:
bbox AP:41.78, 36.01, 30.62
bev  AP:2.80, 2.73, 2.16
3d   AP:0.75, 0.69, 0.54
aos  AP:17.11, 15.19, 12.73
Car AP@0.70, 0.50, 0.50:
bbox AP:41.78, 36.01, 30.62
bev  AP:12.02, 11.42, 9.60
3d   AP:8.11, 7.81, 6.50
aos  AP:17.11, 15.19, 12.73

Car coco AP@0.50:0.05:0.95:
bbox AP:30.56, 26.72, 23.30
bev  AP:3.65, 3.54, 2.94
3d   AP:2.00, 1.93, 1.58
aos  AP:12.53, 11.22, 9.69

Maybe the evaluation script has some problem. But I did not changed the eval code before. That's kind of strange. I will debug and check the eval code. _____update__ the evaluation script is the original ./second/utils/eval/py without change. I got the result with the pretrianed model CLOCs_SecCas_pretrianed as follows:

000001.txt 
Car 0.00 0 1.87 388.15 183.56 424.89 204.88 1.55 1.65 4.04 -15.85 2.42 56.49 1.60 0.37
Car 0.00 0 1.87 388.51 183.50 422.86 203.35 1.55 1.65 4.04 -17.05 2.47 60.51 1.60 0.36
Car 0.00 0 1.87 387.73 183.63 427.21 206.65 1.55 1.65 4.04 -14.64 2.36 52.47 1.60 0.36
Car 0.00 0 1.87 387.24 183.70 429.91 208.73 1.55 1.65 4.04 -13.44 2.31 48.45 1.60 0.34
Car 0.00 0 1.87 388.83 183.45 421.08 202.02 1.55 1.65 4.04 -18.26 2.53 64.54 1.60 0.33
Car 0.00 0 1.86 386.65 183.79 433.07 211.20 1.55 1.65 4.04 -12.23 2.25 44.42 1.60 0.32
Car 0.00 0 1.86 385.94 183.90 436.83 214.19 1.55 1.65 4.04 -11.03 2.20 40.40 1.60 0.30
Car 0.00 0 1.53 0.00 193.96 1242.00 375.00 1.55 1.65 4.04 0.22 1.68 2.59 1.60 0.29
Car 0.00 0 3.15 0.00 0.00 1242.00 375.00 1.55 1.65 4.04 -3.00 1.69 -0.23 1.60 0.29
Car 0.00 0 0.05 0.00 0.00 1242.00 375.00 1.55 1.65 4.04 3.03 1.62 -0.23 1.60 0.29
Car 0.00 0 1.87 389.11 183.41 419.51 200.85 1.55 1.65 4.04 -19.46 2.58 68.56 1.60 0.29
Car 0.00 0 2.00 0.00 195.31 535.63 375.00 1.55 1.65 4.04 -1.39 1.70 2.99 1.60 0.29
Car 0.00 0 1.86 385.06 184.02 441.38 217.88 1.55 1.65 4.04 -9.82 2.14 36.38 1.60 0.29
Car 0.00 0 3.07 0.00 0.00 1242.00 375.00 1.55 1.65 4.04 -4.60 1.71 0.17 1.60 0.28
Car 0.00 0 0.04 0.00 0.00 1242.00 375.00 1.55 1.65 4.04 5.04 1.60 -0.23 1.60 0.28
Car 0.00 0 -0.06 386.92 199.24 943.29 375.00 1.54 1.67 4.06 0.40 1.79 6.10 -0.00 0.28
Car 0.00 0 2.11 0.00 192.79 391.17 375.00 1.55 1.65 4.04 -2.99 1.74 5.00 1.60 0.28
Car 0.00 0 3.03 0.00 0.00 1242.00 375.00 1.55 1.65 4.04 -6.21 1.73 0.58 1.60 0.28
Car 0.00 0 0.04 0.00 0.00 1242.00 375.00 1.55 1.65 4.04 7.05 1.58 -0.23 1.60 0.27
Car 0.00 0 0.05 368.77 195.69 771.32 353.46 1.54 1.67 4.06 -0.40 1.82 8.11 -0.00 0.27
Car 0.00 0 1.86 383.96 184.18 447.00 222.56 1.55 1.65 4.04 -8.62 2.09 32.36 1.60 0.27

It seems that the detection results is very wrong, instead of the evaluation problem.

[pangsu0613]

Hi @yinggo , Ah, I see, I guess you didn't load the SECOND pretrained model so you are running an un-trained SECOND 3D detecor for fusion. We need to run a pretrained SECOND to generate 3D detection candidates for CLOCs. You need to download the weights from here: https://drive.google.com/drive/u/1/folders/1ScFUWPwzK5_VXb-LYQZuZVkiBj-dTMJ9, file "second_model.zip", I also put this link in the README. Unzip the file, and put the unzipped files (checkpoints.json and voxelnet-30950.tckpt) under directory /CLOCs/model_dir/, you could refer to Preperation, step 2 in the README for more detail.

[yinggo]

@pangsu0613 It well solved my problem. Thanks! Now I can get the same result as you present above. I unzipped the files with the 'second_model' folder. Actually, the files should be directly put in the directory /CLOCs/model_dir/. A very stupid mistake. sorry. Further, I want to try PV-RCNN+Cascade R-CNN as the paper reported to achieve the 92.78%, 85.94% and 83.25% in easy, moderate and hard case. I am not clear about the README, Preperation, step 2, "Prepare the 3D detection candidates, run your 3D detector and save the results in the format that SECOND could read," in which variable should the results of 3D detector be saved and from where they should be read? I am not sure about the corresponding part in the code. Currently, in this code, I use the SECOND pretrained model to generate the 3D detection candidates, instead of directly use the 3D detection candidates. Could you give an instruction or an example for how to save and use the 3D detection candidate?

[Joeless]

Hi~ @pangsu0613 I evaluate it by your "CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt" , and get a lower AP than you can you tell why?

My result: Predict_test: False sparse_shape: [ 41 1600 1408] num_class is : 1 load existing model Restoring parameters from ../model_dir/voxelnet-30950.tckpt load existing model for fusion layer Restoring parameters from ../model_dir/CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt remain number of infos: 3769 Generate output labels... [100.0%][===================>][5.14it/s][16:53>00:00]
generate label finished(3.71/s). start eval: validation_loss: 0.15382743618570338 avg example to torch time: 3.456 ms avg prep time: 72.179 ms Car AP@0.70, 0.70, 0.70: bbox AP:99.33, 91.36, 88.33 bev AP:96.16, 87.57, 84.28 3d AP:89.89, 77.75, 72.10 aos AP:99.18, 90.86, 87.39 Car AP@0.70, 0.50, 0.50: bbox AP:99.33, 91.36, 88.33 bev AP:99.43, 94.05, 91.20 3d AP:99.39, 93.92, 91.00 aos AP:99.18, 90.86, 87.39

[pangsu0613]

Hello @Joeless , for your situation, I guess it is the configuration problem. I suggest testing SECOND without CLOCs and see how it performs, so we could see if it is the SECOND configuration problem. You could simpley comment the line below: https://github.com/pangsu0613/CLOCs/blob/cad14fdc12392b9734d496e5d7782ae3ba200af5/second/pytorch/train.py#L466 this will disable CLOCs and only run the pre-trained SECOND, you could compare this results to the original SECOND result to see if there are the same, then we could move forward and see if there are other issues.

[Joeless]

Hi~ @pangsu0613 Thanks for your answer I comment line 466, and get lower AP than the original SECOND My .config only changed the path based on README.md What's correct congfig? My config is at the end of this comment~ Sorry, the paste of this website is very hard to use

Now result: Predict_test: False sparse_shape: [ 41 1600 1408] num_class is : 1 load existing model Restoring parameters from ../model_dir/voxelnet-30950.tckpt load existing model for fusion layer Restoring parameters from /root/mym/git/CLOCs/model_dir/CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt remain number of infos: 3769 Generate output labels... [100.0%][===================>][5.54it/s][14:01>00:00]
generate label finished(4.47/s). start eval: validation_loss: 0.15382743618570338 avg example to torch time: 1.951 ms avg prep time: 54.293 ms Car AP@0.70, 0.70, 0.70: bbox AP:96.09, 89.33, 84.80 bev AP:94.78, 85.89, 81.46 3d AP:88.27, 74.51, 69.43 aos AP:95.96, 88.86, 84.02 Car AP@0.70, 0.50, 0.50: bbox AP:96.09, 89.33, 84.80 bev AP:98.04, 92.59, 89.76 3d AP:97.96, 92.16, 87.61 aos AP:95.96, 88.86, 84.02

My car.fhd.config:

`model: { second: { voxel_generator { point_cloud_range : [0, -40, -3, 70.4, 40, 1] voxel_size : [0.05, 0.05, 0.1] # original is 0.05,0.05,0.1 max_number_of_points_per_voxel : 5 # original is 5 }

voxel_feature_extractor: {
  module_class_name: "VoxelFeatureExtractorV3"
  num_filters: [16]
  with_distance: false
  num_input_features: 4
}
middle_feature_extractor: {
  module_class_name: "SpMiddleFHD"
  downsample_factor: 8
  num_input_features: 4
}
rpn: {
  module_class_name: "RPNV2"
  layer_nums: [5, 5]
  layer_strides: [1, 2]
  num_filters: [128, 256]
  upsample_strides: [1, 2]
  num_upsample_filters: [256, 256]
  use_groupnorm: false
  num_groups: 32
  num_input_features: 128
}
loss: {
  classification_loss: {
    weighted_sigmoid_focal: {
      alpha: 0.25
      gamma: 2.0
      anchorwise_output: true
    }
  }
  localization_loss: {
    weighted_smooth_l1: {
      sigma: 3.0
      code_weight: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
    }
  }
  classification_weight: 1.0   #original 1.0
  localization_weight: 2.0
}
# Outputs
use_sigmoid_score: true
encode_background_as_zeros: true
encode_rad_error_by_sin: true

use_direction_classifier: true # this can help for orientation benchmark
direction_loss_weight: 0.2 # enough.
use_aux_classifier: false
# Loss
pos_class_weight: 1.0
neg_class_weight: 1.0

loss_norm_type: NormByNumPositives
# Postprocess
post_center_limit_range: [0, -40, -3.0, 70.4, 40, 0.0]
use_rotate_nms: true
use_multi_class_nms: false
nms_pre_max_size: 1000
nms_post_max_size: 100
nms_score_threshold: 0.2
nms_iou_threshold: 0.01

use_bev: false
num_point_features: 4
without_reflectivity: false
box_coder: {
  ground_box3d_coder: {
    linear_dim: false
    encode_angle_vector: false
  }
}
target_assigner: {
  anchor_generators: {
    anchor_generator_range: {
      sizes: [1.6, 3.9, 1.56] # wlh
      anchor_ranges: [0, -40.0, -1.78, 70.4, 40.0, -1.78] # carefully set z center, the original one is -1.78
      rotations: [0, 1.57] # DON'T modify this unless you are very familiar with my code.
      matched_threshold : 0.6
      unmatched_threshold : 0.45
      class_name: "Car"
    }
  }
  sample_positive_fraction : -1
  sample_size : 512
  region_similarity_calculator: {
    nearest_iou_similarity: {
    }
  }
}

} }

train_input_reader: { max_num_epochs : 160 batch_size: 1 prefetch_size : 25 max_number_of_voxels: 16000 # to support batchsize=2 in 1080Ti, original 16000 shuffle_points: true num_workers: 3 groundtruth_localization_noise_std: [1.0, 1.0, 0.5] groundtruth_rotation_uniform_noise: [-0.78539816, 0.78539816] global_rotation_uniform_noise: [-0.78539816, 0.78539816] global_scaling_uniform_noise: [0.95, 1.05] global_random_rotation_range_per_object: [0, 0] # pi/4 ~ 3pi/4 anchor_area_threshold: -1 remove_points_after_sample: true groundtruth_points_drop_percentage: 0.0 groundtruth_drop_max_keep_points: 15 database_sampler { database_info_path: "/root/*/git/CLOCs/second/data/kitti_dbinfos_train.pkl" sample_groups { name_to_max_num { key: "Car" value: 15 } } database_prep_steps { filter_by_min_num_points { min_num_point_pairs { key: "Car" value: 5 } } } database_prep_steps { filter_by_difficulty { removed_difficulties: [-1] } } global_random_rotation_range_per_object: [0, 0] rate: 1.0 }

remove_unknown_examples: false remove_environment: false kitti_info_path: "../git/CLOCs/second/data/kitti_infos_train.pkl" kitti_root_path: "../git/CLOCs/second/data" }

train_config: { optimizer: { adam_optimizer: { learning_rate: { one_cycle: { lr_max: 3e-3 # original 3e-3 moms: [0.95, 0.85] div_factor: 10.0 #original 10 pct_start: 0.4 } } weight_decay: 0.01 # super converge. decrease this when you increase steps. og 0.01 } fixed_weight_decay: true use_moving_average: false } steps: 37120 #112215 #113715 #111360 # 619 50, super converge. increase this to achieve slightly better results original 30950 steps_per_eval: 3712 #7481 # 619 5 save_checkpoints_secs : 1800 # half hour 1800 save_summary_steps : 10 enable_mixed_precision: false # for fp16 training, don't use this. loss_scale_factor : 512.0 clear_metrics_every_epoch: true detection_2d_path: "../git/CLOCs/d2_detection_data/data" }

eval_input_reader: { batch_size: 1 max_num_epochs : 160 prefetch_size : 25 max_number_of_voxels: 16000 shuffle_points: false num_workers: 3 anchor_area_threshold: -1 remove_environment: false kitti_info_path: "../git/CLOCs/second/data/kitti_infos_val.pkl" kitti_root_path: "../git/CLOCs/second/data" }`

[xumengqishuai]

hello,you are very great! could you please help me? my ap=0，I need your help!!! @pangsu0613

[Joeless]

@pangsu0613 Hi Sorry for ask you again.... Can you help me find the reason of lower AP?