How to train multi-classes at the same time?

[lucasjinreal]

Hi, If I want train it to detect multi classes, what should I change?

model: {
  second: {
    voxel_generator {
      point_cloud_range : [0, -40, -3, 70.4, 40, 1]
      # point_cloud_range : [0, -32.0, -3, 52.8, 32.0, 1]
      voxel_size : [0.2, 0.2, 0.4]
      max_number_of_points_per_voxel : 35
    }

    num_class: 1
    voxel_feature_extractor: {
      module_class_name: "VoxelFeatureExtractor"
      num_filters: [32, 128]
      with_distance: false
    }
.....

[traveller59]

I will upload a sample config. you can see the following example first.

model: {
  second: {
    ...
    num_class: 3

    target_assigner: {
      anchor_generators: {
        # fixed-size anchor generator for EVERY class.
      anchor_generators: { # cars
        anchor_generator_range: {
          sizes: [1.6, 3.9, 1.56] # wlh
          anchor_ranges: [0, -40, -1.78, 70.4, 40, -1.78] # carefully set z center
          rotations: [0, 1.57] # DON'T modify this unless you are very familiar with my code.
          matched_threshold : 0.6
          unmatched_threshold : 0.45
        }
      }
      anchor_generators: { # cyclist
        anchor_generator_stride: {
          sizes: [0.6, 1.76, 1.73] # wlh
          strides: [0.2, 0.2, 0.0] # if generate only 1 z_center, z_stride will be ignored
          offsets: [0.1, -19.9, -1.465] # origin_offset + strides / 2
          rotations: [0, 1.57]
          matched_threshold : 0.5
          unmatched_threshold : 0.35
        }
      }
      anchor_generators: { # pedestrian
        anchor_generator_stride: {
          sizes: [0.6, 0.8, 1.73] # wlh
          strides: [0.2, 0.2, 0.0] # if generate only 1 z_center, z_stride will be ignored
          offsets: [0.1, -19.9, -1.465] # origin_offset + strides / 2
          rotations: [0, 1.57]
          matched_threshold : 0.5
          unmatched_threshold : 0.35
        }
      }
      ...
    }
  }
}

train_input_reader: {
  # the order of class_names MUST match order of anchor generators
  class_names: ["Car", "Cyclist", "Pedestrian"]
  ...
  database_sampler {
    database_info_path: "/media/yy/960evo/datasets/kitti/kitti_dbinfos_train.pkl"
    sample_groups {
      name_to_max_num {
        key: "Car"
        value: 8
      }
    }
    sample_groups {
      name_to_max_num {
        key: "Pedestrian"
        value: 4
      }
    }
    sample_groups {
      name_to_max_num {
        key: "Cyclist"
        value: 4
      }
    }
    database_prep_steps {
      filter_by_min_num_points {
        min_num_point_pairs {
          key: "Pedestrian"
          value: 5
        }
        min_num_point_pairs {
          key: "Cyclist"
          value: 5
        }
      }
    }
    database_prep_steps {
      filter_by_difficulty {
        removed_difficulties: [-1]
      }
    }
    global_random_rotation_range_per_object: [0, 0]
    rate: 1.0
  }
  ...
}

eval_input_reader: {
  class_names: ["Car", "Cyclist", "Pedestrian"]
  ...
}

[lucasjinreal]

@traveller59 Thanks for you advice. I shall try it out

[zlfclzt1]

Any update for multi-class detection?

[YangHe712]

@traveller59 In order to train multi classes(I only tried 3, car, van, truck), I go through the KITTI dataset and get the average dimensions wlh for each class. The problem now is, if I only change the num_class, class_names and the parameters sizes in the anchor_generator_range of each class, the training accuracy is still quite low (car is OK with AP 0.80+, but van is ~0.4) after 150000+ steps, especially for the Truck which remains 0 all the time. So, I am wondering is there any other parameter that you suggest me to modify? I am not that familiar with your code.

thanks for your suggestions in advance

[kxhit]

I tried to train the model with people.config, but the result is not good. Waiting for multi-class detection.

[MadhavEsDios]

Has anyone tried out the sample config ?

[traveller59]

@MadhavEsDios there is a problem in target assign when training with multi-class, lots of people and cyclist anchors are assigned to car target. expected behavior is every class only be assigned to one kind of anchors. I need time to fix this.

[MadhavEsDios]

Thank you so much @traveller59 for looking into this. So when I was comparing the config files for both classes i.e cars and ped-cycle, I found that apart from the anchor-generator, there were some additional differences namely :-> point_cloud_range, voxel_size and layer_strides. Is this a possible reason for the issues ?

[traveller59]

@MadhavEsDios When add new class to model, you need to change:

1. model.second.num_class
2. add new anchor_generators
3. train_input_reader.class_names
4. eval_input_reader.class_names (I will move this to the model proto in future.)

I have fixed the assign problem by assign targets independently for each class and trained a multi-class (car, ped, cyc, van) model:

Car AP@0.70, 0.70, 0.70:
bbox AP:90.84, 89.53, 88.40
bev  AP:90.09, 86.85, 86.83
3d   AP:88.35, 77.96, 76.03
aos  AP:90.80, 89.19, 87.70
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:83.99, 70.85, 64.79
bev  AP:80.24, 65.75, 60.44
3d   AP:76.89, 59.41, 57.49
aos  AP:83.75, 69.74, 63.86
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:68.37, 62.37, 59.95
bev  AP:62.24, 57.79, 52.19
3d   AP:58.00, 51.32, 48.36
aos  AP:60.23, 54.72, 52.04
Van AP@0.70, 0.70, 0.70:
bbox AP:47.23, 41.50, 35.88
bev  AP:47.41, 41.47, 36.03
3d   AP:41.79, 33.79, 28.65
aos  AP:47.20, 41.41, 35.31

This model use a slightly different architecture and will be released soon. This assign problem will be fixed after the PointPillars authors send a merge pull request to this repo.

[MadhavEsDios]

Thanks a lot @traveller59, looking forward to the new commit.

[chowkamlee81]

Could you please upload sample.config file to train simulatanously for multiclass detection

[lucasjinreal]

@traveller59 Any updates on this?

[traveller59]

@chowkamlee81 there is a sample config all.fhd.config in this project. @jinfagang The people detection need small receptive field. The performance of people detection in car-like config leads to bad performance. we may need a other head from sparse feature extractor for people detection, or a RCNN structure, point cloud of pedestrian can be scaled before get into RCNN. There is another strange thing: I have submit people detection network result to KITTI test server (for my master thesis) and get bad test performance (a little worse than SECOND v1), but the val performance is better than v1.

[kargarisaac]

@traveller59 Is there any pretrained model for pointpillars? It would be great if there is a single model for all classes.

[abhigoku10]

@traveller59 i have following queries

1. can you give a small description of what does the function anchor_generator_stride() and anchor_generator_range() do
2. for custom dataset for custom class should we have to use anchor_generator_range() with values sizes: [1.6, 3.9, 1.56] and anchor_ranges[], is it default since the custom class object is small object thanks in advance