How to get the PGD error of CROWN_IBP?

[DOFYPXY]

Hi,

I try to use attack.py to test the PGD error of the SOTA models on MNIST, by running

python3 attack.py --config config/mnist_dm-large_0.4.json

The epsilon_eval=0.3. I got the result

eps = 0.1
on IBP_large
[TOTAL] clean err: 0.0165 | PGD err: 0.0207
eps = 0.2
on IBP_large
[TOTAL] clean err: 0.0165 | PGD err: 0.0245
eps = 0.30000000000000004
on IBP_large
[TOTAL] clean err: 0.0165 | PGD err: 0.0272
eps = 0.4
on IBP_large
[TOTAL] clean err: 0.0165 | PGD err: 0.0335

The PGD error is much smaller than that reported on paper, which is about 0.0605. (There must be something wrong in my testing method).

I'm very confused about it. Could you please tell me how to test the PGD error correctly? Thank you.

[huanzhang12]

You should not use the attack.py in this repository. That file is not compatible with the latest code and should actually be removed. You need to implement PGD attack yourself to conduct the attack, or using any existing toolbox.