Assertion error using DeepPoly to verify other trained model

[JacksonZyy]

Hi,

I want to use DeepPoly to verify a Pytorch pretrained CNN (the model is converted to ONNX for verification). The network architecture is defined as followed:

layers = []
layers.append(nn.Conv2d(in_channels=self.input_features,\
                        out_channels=self.middle_features,\
                        kernel_size=self.kernel_size,\
                        padding=self.padding,\
                        bias=False))
layers.append(nn.ReLU(inplace=True))
for _ in range(self.num_conv_layers-2):
                         layers.append(nn.Conv2d(in_channels=self.middle_features,\
                        out_channels=self.middle_features,\
                        kernel_size=self.kernel_size,\
                        padding=self.padding,\
                        bias=False))
            layers.append(nn.BatchNorm2d(self.middle_features))
            layers.append(nn.ReLU(inplace=True))
layers.append(nn.Conv2d(in_channels=self.middle_features,\
                        out_channels=self.output_features,\
                        kernel_size=self.kernel_size,\
                        padding=self.padding,\
                        bias=False))

Which contains the Batchnormalization layers during the training process. But for verification, I actually set the model in evaluation mode, and this should remove the batch normalization layers and replace it by an affine layer that applies the learned normalization. But I still have AssertionError Operations of type BatchNormalization are not yet supported. I am wondering why it would give such an error with batch normalization layers removed already?

Thanks, Yuyi

[GgnDpSngh]

Hi there,

It seems that your network in the evaluation mode still contains Batch normalization. If you think that the results of your network are not affected by this layer, I can commit a patch to parse but ignore this layer.

Cheers, Gagandeep Singh

[JacksonZyy]

Hi there,

It seems that your network in the evaluation mode still contains Batch normalization. If you think that the results of your network are not affected by this layer, I can commit a patch to parse but ignore this layer.

Cheers, Gagandeep Singh

Sure, that would be great. Thank you so much for your help! Can you notify me once complete the patch?

Thanks, Yuyi

[GgnDpSngh]

Hi Yuyi,

I have pushed a fix. Let me know if it works. If not, please send me a toy version of your network so that I can test before committing the next fix.

Cheers, Gagandeep Singh

[JacksonZyy]

Hi Yuyi,

I have pushed a fix. Let me know if it works. If not, please send me a toy version of your network so that I can test before committing the next fix.

Cheers, Gagandeep Singh

Hi Gagandeep,

Thank you so much for your help. But I got the following error during execution:

    eran = ERAN(model, is_onnx=is_onnx)
  File "/Verifier_nn/ERAN/tf_verify/eran.py", line 40, in __init__
    operations, resources = translator.translate()
  File "/Verifier_nn/ERAN/tf_verify/onnx_translator.py", line 409, in translate
    filters, bias, image_shape, strides, pad_top, pad_left, kernel_shape = self.conv_resources(node)
  File "/Verifier_nn/ERAN/tf_verify/onnx_translator.py", line 651, in conv_resources
    bias = self.constants_map[node.input[2]]
IndexError: list index (2) out of range

My network is actually for image denoising, not suitable for MNIST/CIFAR, so modifications are required in the main.py to use ERAN to over-approximate my network. I will directly remove the BN layers from my model and try it out on ERAN again. In this way, it won't require ERAN to handle BN layers. Again, really appreciate your help!

Best regards, Yuyi

[GgnDpSngh]

Hi Yuyi,

The error seems to be due to the convolutional layers not having bias, is it true for your networks?

Cheers, Gagandeep Singh

[JacksonZyy]

Hi Yuyi,

The error seems to be due to the convolutional layers not having bias, is it true for your networks?

Cheers, Gagandeep Singh

It is a pretrained model from others, I checked just now, indeed they don't have bias (the layers are created with bias=False) If so, I think I would still see this error even with BN layers removed. Any suggestion how to handle it?

Thanks, Yuyi

[GgnDpSngh]

I will push a fix to handle this and let you know, in that case, you do not need to remove BN layers.

Cheers, Gagandeep Singh

[GgnDpSngh]

Hi Yuyi,

I have pushed a fix, let me know if it works.

Cheers, Gagandeep Singh

[JacksonZyy]

Hi Yuyi,

I have pushed a fix, let me know if it works.

Cheers, Gagandeep Singh

Yes, it is working now!! Thank you! BTW, the network has 105846048 neurons, can ERAN scale up to handle such a large network?

Best regards, Yuyi

[GgnDpSngh]

Hi Yuyi,

The runtime is a function of the network size and the property you are trying to verify. There are results on verifying ImageNet based VGG16 with DeepPoly against the brightening attack (see Table 4 in https://arxiv.org/abs/2004.05511). If the CPU version DeepPoly is too slow for your network and property, you could try the GPU version that we provide (see Table 2 and 3 in https://files.sri.inf.ethz.ch/website/papers/christophgpu.pdf for speedups). For this, you could switch to ERAN and ELINA version from May (the API changed recently and the GPU version is not compatible yet but we will update it in the coming days) and in the config.py file, change the following line:

https://github.com/eth-sri/eran/blob/fd2a6118f8b48e56e6266af3d1ce6c8374c9164d/tf_verify/config.py#L50

to

device = Device.CUDA

Cheers, Gagandeep Singh

[JacksonZyy]

Hi Yuyi,

The runtime is a function of the network size and the property you are trying to verify. There are results on verifying ImageNet based VGG16 with DeepPoly against the brightening attack (see Table 4 in https://arxiv.org/abs/2004.05511). If the CPU version DeepPoly is too slow for your network and property, you could try the GPU version that we provide (see Table 2 and 3 in https://files.sri.inf.ethz.ch/website/papers/christophgpu.pdf for speedups). For this, you could switch to ERAN and ELINA version from May (the API changed recently and the GPU version is not compatible yet but we will update it in the coming days) and in the config.py file, change the following line:

https://github.com/eth-sri/eran/blob/fd2a6118f8b48e56e6266af3d1ce6c8374c9164d/tf_verify/config.py#L50

to

device = Device.CUDA

Cheers, Gagandeep Singh

Noted with thanks! I will close the issue now.

Best regards, Yuyi