关于ANN2SNN 转换VGG16net时出现的问题

[hanzh0816]

我在实现 将ann版本的vgg16网络转换为snn时遇到了严重的测试准确率下降问题（数据集为mnist，ann-vgg测试准确率98%左右，而转换后只有10%左右，时间步从T=10一直到T=200都尝试过，没有大幅提升），我已经做了conv层和bn层融合，但效果同上，请问您遇到过这个问题吗，或者能否给我提供一些帮助呢

[hanzh0816]

补充：VGG16中用的AvgPool2d，转换方式RobustNorm和MaxNorm都尝试过

[fangwei123456]

打印一下转换前的SNN和转换后的ANN网络结构，然后检查一下第一个IF神经元输出频率和ReLU的输出，看看差异

[DingJianhao]

如果需要，请在问题中附上提供您的模型

[hanzh0816]

这是我的模型网络结构，

[DingJianhao]

I can't see your conversion code and model

[hanzh0816]

sorry,刚才出了点问题，附上我的网络结构和转换后的部分snn网络结构

[hanzh0816]

vgg16的代码如下：

class VGGnet(nn.Module):
    def __init__(self, architecture=VGG_16_AVG, in_channels=1, num_classes=10):
        super(VGGnet, self).__init__()
        self.in_channels = in_channels
        self.num_classes = num_classes
        self.conv_layer = self.create_conv_layers(architecture)
        self.fc = nn.Sequential(
            nn.Flatten(),
            nn.Linear(512 * 4 * 4, 4096),
            nn.ReLU(),
            nn.Linear(4096, 4096),
            nn.ReLU(),
            nn.Linear(4096, self.num_classes)
        )

    def forward(self, x):
        x = self.conv_layer(x)
        x = self.fc(x)
        # x = F.log_softmax(x, dim=1)
        x = F.softmax(x, dim=1)
        return x

    def create_conv_layers(self, architecture=VGG_16_AVG):
        net = nn.Sequential()
        num = 0
        in_channels = self.in_channels

        for x in architecture:
            num += 1
            if type(x) == int:
                out_channels = x
                net.add_module(  # add conv layer
                    'conv' + str(num),
                    nn.Conv2d(
                        in_channels=in_channels,
                        out_channels=out_channels,
                        kernel_size=(3, 3),
                        stride=(1, 1),
                        padding=(1, 1)
                    )
                )

                net.add_module(  # add batch_norm layer
                    'bn' + str(num),
                    nn.BatchNorm2d(x)
                )

                net.add_module(
                    'activation_layer' + str(num),
                    nn.ReLU()
                )

                in_channels = x

            elif x == "A":
                net.add_module(
                    'avg_pool' + str(num),
                    nn.AvgPool2d(kernel_size=(2, 2), stride=(2, 2))
                )

            elif x == "M":
                net.add_module(
                    'max_pool' + str(num),
                    nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2))
                )

        return net

    def fused_model(self, architecture=VGG_16_AVG):
        new_model = nn.Sequential()
        num = 0
        for x in architecture:
            num += 1
            if type(x) == int:
                conv = getattr(self.conv_layer, 'conv' + str(num))
                bn = getattr(self.conv_layer, 'bn' + str(num))
                relu = getattr(self.conv_layer, 'activation_layer' + str(num))
                fused_conv = fuse(conv, bn)
                new_model.add_module(
                    'fused_conv' + str(num),
                    fused_conv
                )
                new_model.add_module(
                    'activation_layer' + str(num),
                    relu
                )
            elif x == "A":
                new_model.add_module(
                    'avg_pool' + str(num),
                    getattr(self.conv_layer, 'avg_pool' + str(num))
                )

            elif x == "M":
                new_model.add_module(
                    'max_pool' + str(num),
                    getattr(self.conv_layer, 'max_pool' + str(num))
                )

        new_model.add_module('fc', self.fc)
        return new_model

[DingJianhao]

我感觉create_conv_layers建出来的模型应该没问题啊

[hanzh0816]

我感觉create_conv_layers建出来的模型应该没问题啊

是的，这就是我困惑的地方， #259 里提到的代码跟我的也没什么区别

[Lyu6PosHao]

Hello, @Alex-Hanzh Maybe I could help you if your problem hasn't been solved. First, I tried your ann model but it reported an error, which is because _fc.(1)Linear.inchannels is not proper. It should be 512 rather than 8192. Then, I trained the ann model and got 98% acccuracy. And then I converted it and evaluated the snn model (the parameters: mode='max', T=100). And finally I got 95.2% accuracy.

I admit that T should be large when the model is deep. That's why I set T=100. If you think inference latency is too high, you could try other advanced converting methods.

Here are my outputs of 100 time steps val of snn model:

Hope it is helpful.

[hanzh0816]

@Lyu6PosHao really thanks for your help. I'm trying other alternative input image size and then forgot to modify the fc.(1)Linear.in_channels parameter back to the correct one. thanks for your correction, the model should be fine after modifing it.

as you mentioned the latency will be higher when the model is deeper. I also tried ResNet and other models，all have the same performence. I haven't found a solution to the high latency, do you have any good suggestions？

[Lyu6PosHao]

@Alex-Hanzh 很高兴能有帮助。 要降低延迟的话，可以选择Converter(mode='99%')这种RobustNorm的方法，不过会以损失部分accuracy为代价。具体的输出数据，可以参考issue #289。 如果要追求更好的latency和accuracy，就需要使用论文中的更先进的方法，比如：https://github.com/DingJianhao/OptSNNConvertion-RNL-RIL https://github.com/putshua/SNN_conversion_QCFS 这些方法目前并未在本框架中实现，或许以后我们会将它们嵌入本框架。