Bad results SP-NAS

[vanessasidrim]

I got very low results when running SP-NAS.

The results: {"mAP": 0.16827491265219735, "AP50": 0.5123207922582289, "AP_small": 0.09587199084704223, "AP_medium": 0.24805576528294956, "AP_large": 0.23729036060108377}

I would like to know what your results were in running this network and I would also like to know how I can use the architecture generated after the discovery process

I used the configuration similar to that reported in the network paper:

general:
      parallel_search: True
      parallel_fully_train: True

      pipeline: [serial, parallel, fullytrain]
      serial:
            pipe_step:
                   type: SearchPipeStep
      search_algorithm:
            type: SpNasS
            max_sample: 20
            objective_keys: mAP

      search_space:
            type: SearchSpace
            hyperparameters:
                  key: network.backbone.code
                  type: CATEGORY
                  range: ['111-2111-211111111111111111111111-211']

      model:
             model_desc:
                  type: FasterRCNN
                  weight_file: /vega/cache/models/torch_rpn.pth
                  backbone:
                        type: SerialBackbone
                        weight_file:  /vega/cache/models/nvidia_resnext101-32x4d_200821.pth
                  neck:
                        type: FasterRCNN
                        weight_file:  /vega/cache/models/torch_rpn.pth
                  num_classes: 91

      trainer:
            type: Trainer
                  epochs: 3
                  num_workers: 24
            get_train_metric_after_epoch: False
            model_statistics: False
            load_checkpoint: `False
            is_detection_trainer: True
                  optimizer:
                        type: SGD
                        params:
                              lr: 0.02
                              momentum: 0.9
                              weight_decay: !!float 1e-4
                  lr_scheduler:
                        type: WarmupScheduler
                        by_epoch: False
                        params:
                              warmup_type: linear
                              warmup_iters: 5000
                              warmup_ratio: 0.001
                        after_scheduler_config:
                              type: MultiStepLR
                              by_epoch: True
                              params:
                                    milestones: [3]
                                    gamma: 0.1
                  loss:
                        type: SumLoss
                        metric:
                              type: coco
                        params:
                              anno_path: /vega/cache/datasets/COCO2017/annotations/instances_val2017.json

            dataset:
                  type: CocoDataset
                  common:
                        batch_size: 2
                        data_root: /vega/cache/datasets/COCO2017
                        img_prefix: "2017"
                        ann_prefix: instances
                        transforms:
                              -   type: PolysToMaskTransform
                              -   type: PILToTensor

      parallel:
            pipe_step:
                  type: SearchPipeStep
            search_algorithm:
                  type: SpNasP

            search_space:
                  type: SearchSpace
                  hyperparameters:
                        -   key: network.neck.code
                        type: CATEGORY
                        range: [[1, 0, 2, 1, 2]]

            model:
                  model_desc:
                  type: FasterRCNN
                  backbone:
                        type: SerialBackbone
                  neck:
                        type: ParallelFPN
                  num_classes: 91

            trainer:
                  ref: serial.trainer

            dataset:
                  ref: serial.dataset
                  batch_size: 1

      fullytrain:
            pipe_step:
                  type: TrainPipeStep
                  models_folder: "{local_base_path}/output/parallel/"
                  pretrained_folder: "{local_base_path}/output/parallel/"

            trainer:
                  ref: serial.trainer
                  epochs: 24

            dataset:
                  ref: serial.dataset

[zhangjiajin]

Please provide a complete log to help us identify the problem.

The SP-NAS algorithm performance of the current Vega version is not very good. The efficiency of the original SP-NAS algorithm is low, we are refactoring this algorithm to improve the search efficiency. Before fully train, we need to use imagenet data to fine tune the backbone. This step is missing in the current configuration file. We will release an updated version before July 30.

Before we add the finetune step to the configuration file, you can try running the pre-refactoring version 1.2.0.

https://github.com/huawei-noah/vega/tree/v1.2.0/vega/algorithms/nas/sp_nas https://github.com/huawei-noah/vega/releases/tag/v1.2.0