BR-IDL / PaddleViT

:robot: PaddleViT: State-of-the-art Visual Transformer and MLP Models for PaddlePaddle 2.0+
https://github.com/BR-IDL/PaddleViT
Apache License 2.0
1.22k stars 318 forks source link

LabelSmoothingCrossEntropyLoss增加可选参数 #83

Closed RangeKing closed 2 years ago

RangeKing commented 2 years ago

Describe your feature request LabelSmoothingCrossEntropyLoss增加可选参数 losses.py中(以DeiT为例)LabelSmoothingCrossEntropyLoss可选参数较少 https://github.com/BR-IDL/PaddleViT/blob/dd437d454501943929f29d3e85760fb8e577b284/image_classification/DeiT/losses.py#L21-L46

Describe the reference code or paper

Describe the possible solution 直接调用paddle.nn.functional.cross_entropy计算,可以设置更多参数。 调用前先利用paddle.nn.functional.one_hot将标签转为on-hot形式,再用 paddle.nn.functional.label_smooth将标签平滑,最后将paddle.nn.functional.cross_entropy的soft_label设为True即可实现。 代码如下:

class LabelSmoothingCrossEntropyLoss(nn.Layer):
    def __init__(self,
                 smoothing=0.1,
                 weight=None,
                 ignore_index=-100,
                 reduction='mean',
                 soft_label=True,
                 axis=-1,
                 use_softmax=True,
                 name=None):
        super(LabelSmoothingCrossEntropyLoss, self).__init__()
        assert 0 <= smoothing < 1.0
        self.smoothing = smoothing
        self.weight = weight
        self.reduction = reduction
        self.ignore_index = ignore_index
        self.soft_label = soft_label
        self.axis = axis
        self.use_softmax = use_softmax
        self.name = name

    def forward(self, input, label):
        label = paddle.nn.functional.one_hot(label, num_classes=input.shape[1])
        label = paddle.nn.functional.label_smooth(label, epsilon=self.smoothing)        
        ret = paddle.nn.functional.cross_entropy(
            input,
            label,            
            weight=self.weight,
            ignore_index=self.ignore_index,
            reduction=self.reduction,
            soft_label=self.soft_label,
            axis=self.axis,
            use_softmax=self.use_softmax,
            name=self.name)
        return ret

目前,经过简单测试结果和现有方法计算结果一致。

Additional context Paddle ViT课程训练ResNet18作业中发现,过拟合比较严重,所以想尝试利用Label Smoothing方法缓解。但是搜索paddle api官方文档后发现没有专门的LabelSmoothingCrossEntropyLoss,利用paddle现成的one_hot和label_smooth函数实现了一下。

xperzy commented 2 years ago

Good Work! Would you like to open an PR to merge your implementation? BTW, since this is an useful component, please add an unittest with your PR. Thanks!

RangeKing commented 2 years ago

Good Work! Would you like to open an PR to merge your implementation? BTW, since this is an useful component, please add an unittest with your PR. Thanks!

Thanks for the reply! I'll give it a try.