Apply CRB active learning to other models

yux326 commented 1 year ago

Hello! Thank you for your amazing work! Recently I am trying to apply CRB to other models since PV RCNN requires to much GPU memories. I chose Pointpillar and kitti dataset but it comes out this error:

Traceback (most recent call last): File "train.py", line 313, in main() File "train.py", line 255, in main dist_train=dist_train File "/home/CRB-active-3Ddet/tools/train_utils/train_active_utils.py", line 313, in train_model_active accumulated_iter=accumulated_iter File "../pcdet/utils/active_training_utils.py", line 271, in select_active_labels selected_frames = strategy.query(leave_pbar, cur_epoch) File "../pcdet/query_strategies/crb_sampling.py", line 80, in query preddicts, = self.model(unlabelled_batch) File "/opt/conda/envs/crb/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "../pcdet/models/detectors/pointpillar.py", line 21, in forward pred_dicts, recall_dicts = self.post_processing(batch_dict) File "../pcdet/models/detectors/detector3d_template.py", line 232, in post_processing full_cls_scores = batch_dict['full_cls_scores'][batch_mask] KeyError: 'full_cls_scores'

I will put my config on the next comment.

Is there anything wrong in my config file？Or is it possible to apply CRB to Pointpillar? Thanks so much and looking forward to your reply.

yux326 commented 1 year ago

My config file is like: CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG: _BASECONFIG: cfgs/dataset_configs/kitti_dataset.yaml POINT_CLOUD_RANGE: [0, -39.68, -3, 69.12, 39.68, 1] DATA_PROCESSOR:

NAME: mask_points_and_boxes_outside_range REMOVE_OUTSIDE_BOXES: True
NAME: shuffle_points SHUFFLE_ENABLED: { 'train': True, 'test': False }
NAME: transform_points_to_voxels VOXEL_SIZE: [0.16, 0.16, 4] MAX_POINTS_PER_VOXEL: 32 MAX_NUMBER_OF_VOXELS: { 'train': 16000, 'test': 40000 } DATA_AUGMENTOR: DISABLE_AUG_LIST: ['placeholder'] AUG_CONFIG_LIST:
- NAME: gt_sampling USE_ROAD_PLANE: False DB_INFO_PATH:
  - kitti_dbinfos_train.pkl PREPARE: { filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'], filter_by_difficulty: [-1], }
    
    SAMPLE_GROUPS: ['Car:15','Pedestrian:15', 'Cyclist:15'] NUM_POINT_FEATURES: 4 DATABASE_WITH_FAKELIDAR: False REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0] LIMIT_WHOLE_SCENE: False
- NAME: random_world_flip ALONG_AXIS_LIST: ['x']
- NAME: random_world_rotation WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]
- NAME: random_world_scaling WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL: NAME: PointPillar

VFE:
    NAME: PillarVFE
    WITH_DISTANCE: False
    USE_ABSLOTE_XYZ: True
    USE_NORM: True
    NUM_FILTERS: [64]

MAP_TO_BEV:
    NAME: PointPillarScatter
    NUM_BEV_FEATURES: 64

BACKBONE_2D:
    NAME: BaseBEVBackbone
    LAYER_NUMS: [3, 5, 5]
    LAYER_STRIDES: [2, 2, 2]
    NUM_FILTERS: [64, 128, 256]
    UPSAMPLE_STRIDES: [1, 2, 4]
    NUM_UPSAMPLE_FILTERS: [128, 128, 128]

DENSE_HEAD:
    NAME: AnchorHeadSingle
    CLASS_AGNOSTIC: False

    USE_DIRECTION_CLASSIFIER: True
    DIR_OFFSET: 0.78539
    DIR_LIMIT_OFFSET: 0.0
    NUM_DIR_BINS: 2

    ANCHOR_GENERATOR_CONFIG: [
        {
            'class_name': 'Car',
            'anchor_sizes': [[3.9, 1.6, 1.56]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-1.78],
            'align_center': False,
            'feature_map_stride': 2,
            'matched_threshold': 0.6,
            'unmatched_threshold': 0.45
        },
        {
            'class_name': 'Pedestrian',
            'anchor_sizes': [[0.8, 0.6, 1.73]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-0.6],
            'align_center': False,
            'feature_map_stride': 2,
            'matched_threshold': 0.5,
            'unmatched_threshold': 0.35
        },
        {
            'class_name': 'Cyclist',
            'anchor_sizes': [[1.76, 0.6, 1.73]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-0.6],
            'align_center': False,
            'feature_map_stride': 2,
            'matched_threshold': 0.5,
            'unmatched_threshold': 0.35
        }
    ]

    TARGET_ASSIGNER_CONFIG:
        NAME: AxisAlignedTargetAssigner
        POS_FRACTION: -1.0
        SAMPLE_SIZE: 512
        NORM_BY_NUM_EXAMPLES: False
        MATCH_HEIGHT: False
        BOX_CODER: ResidualCoder

    LOSS_CONFIG:
        LOSS_WEIGHTS: {
            'cls_weight': 1.0,
            'loc_weight': 2.0,
            'dir_weight': 0.2,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
        }

POST_PROCESSING:
    RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
    SCORE_THRESH: 0.1
    OUTPUT_RAW_SCORE: False

    EVAL_METRIC: kitti

    NMS_CONFIG:
        MULTI_CLASSES_NMS: False
        NMS_TYPE: nms_gpu
        NMS_THRESH: 0.01
        NMS_PRE_MAXSIZE: 4096
        NMS_POST_MAXSIZE: 500

OPTIMIZATION: OPTIMIZER: adam_onecycle LR: 0.003 WEIGHT_DECAY: 0.01 MOMENTUM: 0.9

MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001

LR_WARMUP: False
WARMUP_EPOCH: 1

GRAD_NORM_CLIP: 10

ACTIVE_TRAIN: METHOD: crb AGGREGATION: mean

PRE_TRAIN_SAMPLE_NUMS: 100  
PRE_TRAIN_EPOCH_NUMS: 2
TRAIN_RESUME: False

SELECT_NUMS: 100
SELECT_LABEL_EPOCH_INTERVAL: 40

TOTAL_BUDGET_NUMS: 600

ACTIVE_CONFIG:
    K1: 5
    K2: 3
    BANDWIDTH: 5
    CLUSTERING: kmeans++

zhuoxiao-chen commented 1 year ago

Hi, our framework currently supports two 3D detection models: PV-RCNN and SECONDIoU.

To run Point-RCNN, you need to add some code that passes some key-value pairs in the data_dict that are required for active selection, such as the computed density, full_cls_scores, embedding and so on. Note also that there is no dropout layer in Point-RCNN, so a dropout layer needs to be added to enable stage 2 of CRB.

A good suggestion for you is to go through the code for active selection when using PVRCNN and see what are required in the data_dict/batch_dict.

yux326 commented 1 year ago

Thanks so much for replying me, but I am trying Pointpillar instead of Point RCNN, Do you mean that only models with dropout layers can support CRB, otherwise I may have to make some change on the model(e.g. add a dropout layer)?

zhuoxiao-chen commented 8 months ago

Yes, because our stage 2 requires dropout to get hypothesis labels. You may simply add a dropout layer to make it work.

Luoyadan / CRB-active-3Ddet

Apply CRB active learning to other models #13