eth-sri / eran

ETH Robustness Analyzer for Deep Neural Networks
Apache License 2.0
320 stars 103 forks source link

PRIMA does not outperform DeepPoly when adding maxpool layer #113

Open Gting6 opened 2 years ago

Gting6 commented 2 years ago

Hi,

After successfully reproducing the result of CIFAR10 ConvSmall network in Table 2, I tried to use the same config parameter for a modified CIFAR10 ConvSmall network (adding maxpool layer as follows).

image

However, the result of PRIMA is equal to DeepPoly in this setting. I am wondering which parameter should I alter if I have maxpool layer in my neural network?

image

Command for PRIMA: (Same as config in table 2, except for the netname, num_test, mean, and std) python3 . --approx_k True --dataset cifar10 --domain refinepoly --epsilon 0.0000 --from_test 0 --k 3 --max_milp_neurons 100 --n_milp_refine 1 --netname ../net/convSmall_normal_cifar10.onnx --num_tests 100 --partial_milp 2 --s -2 --sparse_n 100 --timeout_final_lp 20.0 --timeout_final_milp 200.0 --timeout_lp 1 --timeout_milp 1 --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225 Command for DeepPoly python3 . --netname ../net/convBig_normal_cifar10.onnx --dataset cifar10 --domain deeppoly --epsilon 0.0000 --num_test 100 --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225