PaddleDetection模型搭建

描述问题/Describe the bug

我想给ssdlite_mobilenet_v3_large添加一个neck模块，也就是FPN，但是给的提示是 ”该模块为注册“

复现/Reproduction

您使用的命令是？/What command or script did you run

python tools/train.py -c=configs/ssd/ssdlite_mobilenet_v3_large_fpn_320_coco.yml -o pretrain_weights='' --eval

您是否更改过代码或配置文件？您是否理解您所更改的内容？还请您提供所更改的部分代码。/Did you make any modifications on the code or config? Did you understand what you have modified? Please provide the codes that you modified.
ssdlite_mobilenet_v3_larg_fpn_320_coco.yml
```
_BASE_: [
'../datasets/coco_detection.yml',
'../runtime.yml',
'_base_/optimizer_1700e.yml',
'_base_/ssdlite_mobilenet_v3_large_fpn_320.yml',
'_base_/ssdlite320_reader.yml',
]
weights: output/ssdlite_mobilenet_v3_large_320_coco/model_final
```

base/ssdlite_mobilenet_v3_large_fpn_320.yml


architecture: SSD
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/MobileNetV3_large_x1_0_ssld_pretrained.pdparams

SSD: backbone: MobileNetV3 ssd_head: SSDHead post_process: BBoxPostProcess neck: MyFPN

MobileNetV3: scale: 1.0 model_name: large conv_decay: 0.00004 with_extra_blocks: true extra_block_filters: [[256, 512], [128, 256], [128, 256], [64, 128]] feature_maps: [14, 17, 18, 19, 20, 21] lr_mult_list: [0.25, 0.25, 0.5, 0.5, 0.75] multiplier: 0.5

SSDHead: use_sepconv: True conv_decay: 0.00004 anchor_generator: steps: [16, 32, 64, 107, 160, 320] aspect_ratios: [[2.], [2., 3.], [2., 3.], [2., 3.], [2., 3.], [2., 3.]] min_ratio: 20 max_ratio: 95 base_size: 320 min_sizes: [] max_sizes: [] offset: 0.5 flip: true clip: true min_max_aspect_ratios_order: false

BBoxPostProcess: decode: name: SSDBox nms: name: MultiClassNMS keep_top_k: 200 score_threshold: 0.01 nms_threshold: 0.45 nms_top_k: 400 nms_eta: 1.0

MyFPN:

说明：就在ssdlite_mobilenet_v3_large_fpn.yml文件中添加了MyFPN模块（和原论文中FPN实现相同，后面会给出代码）

3. 您使用的数据集是？/What dataset did you use?
coco

4. 请提供您出现的报错信息及相关log。/Please provide the error messages or relevant log information.
```bash
Traceback (most recent call last):
  File "tools/train.py", line 171, in <module>
    main()
  File "tools/train.py", line 167, in main
    run(FLAGS, cfg)
  File "tools/train.py", line 118, in run
    trainer = Trainer(cfg, mode='train')
  File "/PaddleDetectionNew/ppdet/engine/trainer.py", line 90, in __init__
    self.model = create(cfg.architecture)
  File "/PaddleDetectionNew/ppdet/core/workspace.py", line 238, in create
    cls_kwargs.update(cls.from_config(config, **kwargs))
  File "/PaddleDetectionNew/ppdet/modeling/architectures/ssd.py", line 61, in from_config
    neck = create(cfg['neck'], **kwargs)
  File "/PaddleDetectionNew/ppdet/core/workspace.py", line 215, in create
    "the module {} is not registered".format(name)
AssertionError: the module MyFPN is not registered

环境/Environment

请问您使用的CUDA/cuDNN的版本号是？/ Please provide the version of CUDA/cuDNN you used. paddlepaddle: 2.2.0, PaddleDetection: 2.3

另附上修改之后的代码

architectures/ssd.py


from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from ppdet.core.workspace import register, create from .meta_arch import BaseArch

all = ['SSD']

@register class SSD(BaseArch): """ Single Shot MultiBox Detector, see https://arxiv.org/abs/1512.02325

Args:
    backbone (nn.Layer): backbone instance
    ssd_head (nn.Layer): `SSDHead` instance
    post_process (object): `BBoxPostProcess` instance
"""

__category__ = 'architecture'
__inject__ = ['post_process']

def __init__(self, backbone, ssd_head, post_process, neck=None, r34_backbone=False):
    super(SSD, self).__init__()
    self.backbone = backbone
    self.ssd_head = ssd_head
    self.post_process = post_process
    self.neck = neck
    self.r34_backbone = r34_backbone
    if self.r34_backbone:
        from ppdet.modeling.backbones.resnet import ResNet
        assert isinstance(self.backbone, ResNet) and \
               self.backbone.depth == 34, \
            "If you set r34_backbone=True, please use ResNet-34 as backbone."
        self.backbone.res_layers[2].blocks[0].branch2a.conv._stride = [1, 1]
        self.backbone.res_layers[2].blocks[0].short.conv._stride = [1, 1]

@classmethod
def from_config(cls, cfg, *args, **kwargs):
    # backbone
    backbone = create(cfg['backbone'])

    # neck
    kwargs = {'input_shape': backbone.out_shape}
    neck = create(cfg['neck'], **kwargs)

    # head
    kwargs = {'input_shape': neck.out_shape}
    ssd_head = create(cfg['ssd_head'], **kwargs)

    return {
        'backbone': backbone,
        'neck': neck,
        "ssd_head": ssd_head,
    }

def _forward(self):
    # Backbone
    body_feats = self.backbone(self.inputs)

    # if self.neck is not None:
    body_feats = self.neck(body_feats)

    # SSD Head
    if self.training:
        return self.ssd_head(body_feats, self.inputs['image'],
                             self.inputs['gt_bbox'],
                             self.inputs['gt_class'])
    else:
        preds, anchors = self.ssd_head(body_feats, self.inputs['image'])
        bbox, bbox_num = self.post_process(preds, anchors,
                                           self.inputs['im_shape'],
                                           self.inputs['scale_factor'])
        return bbox, bbox_num

def get_loss(self, ):
    return {"loss": self._forward()}

def get_pred(self):
    bbox_pred, bbox_num = self._forward()
    output = {
        "bbox": bbox_pred,
        "bbox_num": bbox_num,
    }
    return output

说明：
> 1. __init__函数增加了neck参数
> 2. from_config函数增加了关于neck部分的代码
> 3. _forward函数增加了 body_feats = self.neck(body_feats)
仅在ssd.py中增加了以上代码

> neck/my_fpn.py
```python
import paddle.nn as nn
import paddle.nn.functional as F
from paddle import ParamAttr
from paddle.nn.initializer import XavierUniform

from ppdet.core.workspace import register, serializable
from ppdet.modeling.layers import ConvNormLayer
from ..shape_spec import ShapeSpec

__all__ = ['MyFPN']

@register
@serializable
class MyFPN(nn.Layer):
    """
    BackwardFusion: backward fusion networks

    Args:
        in_channels (list[int]): input channels of each level which can be
            derived from the output shape of backbone by from_config
        out_channel (list[int]): output channel of each level
        spatial_scales (list[float]): the spatial scales between input feature
            maps and original input image which can be derived from the output
            shape of backbone by from_config
        has_extra_convs (bool): whether to add extra conv to the last level.
            default False
        extra_stage (int): the number of extra stages added to the last level.
            default 1
        use_c5 (bool): Whether to use c5 as the input of extra stage,
            otherwise p5 is used. default True
        norm_type (string|None): The normalization type in FPN module. If
            norm_type is None, norm will not be used after conv and if
            norm_type is string, bn, gn, sync_bn are available. default None
        norm_decay (float): weight decay for normalization layer weights.
            default 0.
        freeze_norm (bool): whether to freeze normalization layer.
            default False
        relu_before_extra_convs (bool): whether to add relu before extra convs.
            default False

    """

    def __init__(self,
                 in_channels=[672, 960, 512, 256, 256, 128],
                 out_channel=256,
                 spatial_scales=[0.25, 0.125, 0.0625, 0.03125],
                 norm_type=None,
                 norm_decay=0.,
                 freeze_norm=False):
        super(MyFPN, self).__init__()
        for s in range(extra_stage):
            spatial_scales = spatial_scales + [spatial_scales[-1] / 2.]
        self.in_channels = in_channels
        self.spatial_scales = spatial_scales
        self.out_channel = out_channel
        self.norm_type = norm_type
        self.norm_decay = norm_decay
        self.freeze_norm = freeze_norm

        self.lateral_convs = []
        self.fpn_convs = []
        fan = self.out_channel * 3 * 3
        fpn_out_channel = 256

        for i in range(self.in_channels):
            lateral_name = 'fpn_inner_' + str(i) + '_lateral'
            if self.norm_type is not None:
                lateral = self.add_sublayer(
                    lateral_name,
                    ConvNormLayer(
                        ch_in=self.in_channels[i],
                        ch_out=self.out_channel,
                        filter_size=1,
                        stride=1,
                        norm_type=self.norm_type,
                        norm_decay=self.norm_decay,
                        freeze_norm=self.freeze_norm,
                        initializer=XavierUniform(fan_out=self.in_channels[i])))
            self.lateral_convs.append(lateral)

            fpn_name = 'fpn_sum_' + str(i)
            if self.norm_type is not None:
                fpn_conv = self.add_sublayer(
                    fpn_name,
                    ConvNormLayer(
                        ch_in=fpn_out_channel,
                        ch_out=fpn_out_channel,
                        filter_size=3,
                        stride=1,
                        norm_type=self.norm_type,
                        norm_decay=self.norm_decay,
                        freeze_norm=self.freeze_norm,
                        initializer=XavierUniform(fan_out=fan)))
            self.fpn_convs.append(fpn_conv)

    @classmethod
    def from_config(cls, cfg, input_shape):
        return {
            'in_channels': [i.channels for i in input_shape],
            'spatial_scales': [1.0 / i.stride for i in input_shape],
        }

    def forward(self, body_feats):
        laterals = []
        num_levels = len(body_feats) # len(body_feats) = 6
        for i in range(num_levels):
            laterals.append(self.lateral_convs[i](body_feats[i]))

        # upside + element add
        for i in range(1, num_levels):
            lvl = num_levels - i
            upsample = F.interpolate(
                laterals[lvl],
                scale_factor=2.,
                mode='nearest', )
            laterals[lvl - 1] += upsample

        # 特征融合之后通过3x3/256卷积，防止混叠效应
        fpn_output = []
        for lvl in range(num_levels): # num_levels = 6
            fpn_output.append(self.fpn_convs[lvl](laterals[lvl]))

        return fpn_output

    @property
    def out_shape(self):
        return [
            ShapeSpec(
                channels=self.out_channel, stride=1. / s)
            for s in self.spatial_scales
        ]

my_fpn.py代码是参考官方的fpn.py，进行一定程度的删减。在necks/init.py进行了相应的修改。

ppdet/modeling/necks/__init__.py 有没有写

from . import my_fpn
from .my_fpn import *

ppdet/modeling/necks/__init__.py 有没有写
from . import my_fpn
from .my_fpn import *

做了相应的修改，已经按照官方提供的文档进行编写了，但是还是出现上面的问题觉得很困惑，不知道是不是还有哪个地方遗漏了？

你是不是pip install paddledet，本地修改代码的话记得pip uninstall paddledet然后export PYTHONPATH=pwd:$PYTHONPATH

你是不是pip install paddledet，本地修改代码的话记得pip uninstall paddledet然后export PYTHONPATH=pwd:$PYTHONPATH

我尝试了下面两种方式：

我重新使用 python setup.py install 安装了一下paddledet（我好像忘记安装paddledet了），但还是这个问题，my_fpn没有注册。
您上面提到的pip uninstall paddledet，然后export PYTHONPATH=pwd:$PYTHONPATH，还是不行，同样的问题。

log:

$ pip uninstall paddledet
Found existing installation: paddledet 2.3.0
Uninstalling paddledet-2.3.0:
  Would remove:
    /anaconda3/envs/dxl/lib/python3.7/site-packages/paddledet-2.3.0-py3.7.egg
Proceed (y/n)? y
  Successfully uninstalled paddledet-2.3.0
$ python
Python 3.7.9 (default, Aug 31 2020, 12:42:55) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paddle
>>> print(paddle.__version__)
2.2.0
>>> exit()
$ export PYTHONPATH=pwd:$PYTHONPATH
$ python tools/train.py -c=configs/ssd/ssdlite_mobilenet_v3_large_fpn_320_visdrone.yml  -o pretrain_weights='' --eval
loading annotations into memory...
Done (t=1.42s)
creating index...
index created!
[12/21 11:36:27] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 868, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
W1221 11:36:30.123328 30702 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.0, Runtime API Version: 10.1
W1221 11:36:30.129550 30702 device_context.cc:465] device: 0, cuDNN Version: 7.6.
Traceback (most recent call last):
  File "tools/train.py", line 171, in <module>
    main()
  File "tools/train.py", line 167, in main
    run(FLAGS, cfg)
  File "tools/train.py", line 118, in run
    trainer = Trainer(cfg, mode='train')
  File "/PaddleDetectionNew/ppdet/engine/trainer.py", line 90, in __init__
    self.model = create(cfg.architecture)
  File "/PaddleDetectionNew/ppdet/core/workspace.py", line 238, in create
    cls_kwargs.update(cls.from_config(config, **kwargs))
  File "/PaddleDetectionNew/ppdet/modeling/architectures/ssd.py", line 65, in from_config
    neck = create(cfg['neck'], **kwargs)
  File "/PaddleDetectionNew/ppdet/core/workspace.py", line 215, in create
    "the module {} is not registered".format(name)
AssertionError: the module MyFPN is not registered

你是不是pip install paddledet，本地修改代码的话记得pip uninstall paddledet然后export PYTHONPATH=pwd:$PYTHONPATH

这次我又尝试重新下载PaddleDetection，由于我再之前的PaddleDetection中创建了一个新的git分支，为防止这种情况，所以重新下载然后跑上面的模型，还是为注册MyFPN模块。

建议分享下你的分支，我们帮你debug下

建议分享下你的分支，我们帮你debug下

不好意思，我这边在使用PaddleDetection 2.0/rc版本进行修改看看行不行，没有看到这条消息。这是我新建的仓库：https://github.com/codedlong/PaddleDetection_tmp 还是说我提交一个新分支给你们那边？

建议分享下你的分支，我们帮你debug下

不好意思，我这边在使用PaddleDetection 2.0/rc版本进行修改看看行不行，没有看到这条消息。这是我新建的仓库：https://github.com/codedlong/PaddleDetection_tmp；还是说我提交一个新分支给你们那边？

上面我提到了在PaddleDetection 2.0/rc, paddlepaddle: 2.2.0中添加模块，经过测试，是可以的。训练初始log:

INFO:__main__:iter: 0, lr: 0.001000, 'loss': '51.366745', eta: 0:00:04, batch_cost: 0.00006 sec, ips: 1024562.80916 images/sec
INFO:__main__:iter: 20, lr: 0.001000, 'loss': '38.758072', eta: 2 days, 23:44:28, batch_cost: 3.22916 sec, ips: 19.81938 images/secINFO:__main__:iter: 40, lr: 0.001000, 'loss': '23.677593', eta: 2 days, 4:19:10, batch_cost: 2.35556 sec, ips: 27.16979 images/sec

配置文件

configs/ssd/ssdlite_mobilenet_v3_large_backward.yml


architecture: SSD
use_gpu: true
max_iters: 80000
snapshot_iter: 3000
log_iter: 20
metric: COCO
pretrain_weights: pretrainWeights/MobileNetV3_large_x1_0_ssld_fpn_pretrained
save_dir: output
weights: output/ssdlite_mobilenet_v3_large_fpn/model_final
# 80(label_class) + 1(background)
num_classes: 10

SSD: backbone: MobileNetV3 fpn: BackwardFusion multi_box_head: SSDLiteMultiBoxHead output_decoder: background_label: 0 keep_top_k: 200 nms_eta: 1.0 nms_threshold: 0.45 nms_top_k: 400 score_threshold: 0.01

BackwardFusion: num_chan: 256 max_level: 7 norm_type: bn norm_decay: 0.00004 reverse_out: true

MobileNetV3: scale: 1.0 model_name: large extra_block_filters: [[256, 512], [128, 256], [128, 256], [64, 128]] feature_maps: [5, 7, 8, 9, 10, 11] lr_mult_list: [0.25, 0.25, 0.5, 0.5, 0.75] conv_decay: 0.00004

SSDLiteMultiBoxHead: aspect_ratios: [[2.], [2., 3.], [2., 3.], [2., 3.], [2., 3.], [2., 3.]] base_size: 320 steps: [16, 32, 64, 107, 160, 320] flip: true clip: true max_ratio: 95 min_ratio: 20 offset: 0.5 conv_decay: 0.00004

LearningRate: base_lr: 0.001 schedulers:

!PiecewiseDecay gamma: 0.1 milestones: [54000, 74000]

OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0005 type: L2

TrainReader: inputs_def: image_shape: [3, 320, 320] fields: ['image', 'gt_bbox', 'gt_class'] dataset: !COCODataSet dataset_dir: dataset/VisDrone2019_coco anno_path: annotations/train.json image_dir: train sample_transforms:

!DecodeImage to_rgb: true
!RandomDistort brightness_lower: 0.875 brightness_upper: 1.125 is_order: true
!RandomExpand fill_value: [123.675, 116.28, 103.53]
!RandomCrop allow_no_crop: false
!NormalizeBox {}
!ResizeImage interp: 1 target_size: 320 use_cv2: false
!RandomFlipImage is_normalized: false
!NormalizeImage mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_scale: true is_channel_first: false
!Permute to_bgr: false channel_first: true batch_size: 64 shuffle: true drop_last: true
Number of working threads/processes. To speed up, can be set to 16 or 32 etc.

worker_num: 8

Size of shared memory used in result queue. After increasing worker_num, need expand memsize.

memsize: 8G

Buffer size for multi threads/processes.one instance in buffer is one batch data.

To speed up, can be set to 64 or 128 etc.

bufsize: 32 use_process: true

EvalReader: inputs_def: image_shape: [3, 320, 320] fields: ['image', 'gt_bbox', 'gt_class', 'im_shape', 'im_id'] dataset: !COCODataSet dataset_dir: dataset/VisDrone2019_coco anno_path: annotations/val.json image_dir: val sample_transforms:

!DecodeImage to_rgb: true
!NormalizeBox {}
!ResizeImage interp: 1 target_size: 320 use_cv2: false
!NormalizeImage mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_scale: true is_channel_first: false
!Permute to_bgr: false channel_first: True batch_size: 8 worker_num: 8 bufsize: 32 use_process: false

TestReader: inputs_def: image_shape: [3,320,320] fields: ['image', 'im_id', 'im_shape'] dataset: !ImageFolder anno_path: annotations/instances_val2017.json sample_transforms:

!DecodeImage to_rgb: true
!ResizeImage interp: 1 max_size: 0 target_size: 320 use_cv2: true
!NormalizeImage mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_scale: true is_channel_first: false

!Permute to_bgr: false channel_first: True batch_size: 1


说明：也就是将原来FPN模块改写成了自己书写BackwardFusion模块，数据集换成了Visdrone

具体代码

ppdet/modeling/backbones/backward_fusion.py ppdet
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from collections import OrderedDict import copy from paddle import fluid from paddle.fluid.param_attr import ParamAttr from paddle.fluid.initializer import Xavier from paddle.fluid.regularizer import L2Decay

from ppdet.core.workspace import register from ppdet.modeling.ops import ConvNorm

all = ['BackwardFusion']

@register class BackwardFusion(object): """ Args: num_chan (int): number of feature channels min_level (int): lowest level of the backbone feature map to use max_level (int): highest level of the backbone feature map to use spatial_scale (list): feature map scaling factor has_extra_convs (bool): whether has extral convolutions in higher levels norm_type (str|None): normalization type, 'bn'/'sync_bn'/'affine_channel' norm_decay (float): weight decay for normalization layer weights. reverse_out (bool): whether to flip the output. """ shared = ['norm_type', 'freeze_norm']

def __init__(self,
             num_chan=256,
             min_level=2,
             max_level=6, # 7
             spatial_scale=[1. / 32., 1. / 16., 1. / 8., 1. / 4.],
             has_extra_convs=False,
             norm_type=None,
             norm_decay=0.,
             freeze_norm=False,
             use_c5=True,
             reverse_out=False):
    self.freeze_norm = freeze_norm
    self.num_chan = num_chan
    self.min_level = min_level
    self.max_level = max_level
    self.spatial_scale = spatial_scale
    self.has_extra_convs = has_extra_convs
    self.norm_type = norm_type
    self.norm_decay = norm_decay
    self.use_c5 = use_c5
    self.reverse_out = reverse_out

def _add_downtop_lateral(self, body_name, body_input, down_output):
    lateral_name = 'backward_inner_' + body_name + '_lateral'
    topdown_name = 'backward_downtop_' + body_name
    fan = body_input.shape[1]
    if self.norm_type:
        initializer = Xavier(fan_out=fan)
        lateral = ConvNorm(
            body_input,
            self.num_chan,
            1,
            initializer=initializer,
            norm_type=self.norm_type,
            norm_decay=self.norm_decay,
            freeze_norm=self.freeze_norm,
            name=lateral_name,
            norm_name=lateral_name)

    # 首先对down_output进行下采样
    down_name = 'backward_downsample_' + body_name + '_lateral'
    down_output = ConvNorm(
        down_output,
        self.num_chan,
        filter_size=3,
        stride=2,
        initializer=Xavier(fan_out=fan),
        name=down_name,
        norm_name=down_name)

    return fluid.layers.elementwise_add(lateral, down_output)

def get_output(self, body_dict):
    spatial_scale = copy.deepcopy(self.spatial_scale)
    body_name_list = list(body_dict.keys())
    num_backbone_stages = len(body_name_list) # 6
    self.backward_inner_output = [[] for _ in range(num_backbone_stages)]
    backward_inner_name = 'backward_inner_' + body_name_list[0]
    body_input = body_dict[body_name_list[0]]
    fan = body_input.shape[1]
    if self.norm_type:
        initializer = Xavier(fan_out=fan)
        self.backward_inner_output[0] = ConvNorm(
            body_input,
            self.num_chan,
            1,
            initializer=initializer,
            norm_type=self.norm_type,
            norm_decay=self.norm_decay,
            freeze_norm=self.freeze_norm,
            name=backward_inner_name,
            norm_name=backward_inner_name)
    for i in range(1, num_backbone_stages):
        body_name = body_name_list[i]
        body_input = body_dict[body_name]
        down_output = self.backward_inner_output[i - 1]
        backward_inner_single = self._add_downtop_lateral(body_name, body_input,
                                                     down_output)
        self.backward_inner_output[i] = backward_inner_single
    fpn_dict = {}
    fpn_name_list = []
    for i in range(num_backbone_stages):
        fpn_name = 'backward_' + body_name_list[i]
        fan = self.backward_inner_output[i].shape[1] * 3 * 3
        if self.norm_type:
            initializer = Xavier(fan_out=fan)
            fpn_output = ConvNorm(
                self.backward_inner_output[i],
                self.num_chan,
                3,
                initializer=initializer,
                norm_type=self.norm_type,
                norm_decay=self.norm_decay,
                freeze_norm=self.freeze_norm,
                name=fpn_name,
                norm_name=fpn_name)
        fpn_dict[fpn_name] = fpn_output
        fpn_name_list.append(fpn_name)

    # if self.reverse_out:
    #     fpn_name_list = fpn_name_list[::-1]
    res_dict = OrderedDict([(k, fpn_dict[k]) for k in fpn_name_list])
    return res_dict, spatial_scale


说明：上面书写的代码虽然和前面的my_fpn有区别，但是按照官方给的自己添加模块的教程，PaddleDetection 2.0/rc 和 PaddleDetection 2.3有一些差别，但不至于出现没有注册现象。

ssd.py这个代码没有任何改动。

PaddlePaddle / PaddleDetection