Failed to reproduce the results of attack on ODE?

[thuwzy]

I tried to run attack on ODE. However, the results differ a lot from the reported results in the paper. I got the accuracy of 68% of Linf attack on cifar10 with 8/255. However, the reported result in the paper is 39.86%. Here is my script. Is it anything wrong?

            CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python eval_sde_adv.py --exp ./exp_results --config cifar10.yml \
            -i xxx \
            --t 100 \
            --adv_eps 0.031373 \
            --adv_batch_size 8 \
            --num_sub 64 \
            --domain cifar10 \
            --classifier_name cifar10-wideresnet-28-10 \
            --seed $seed \
            --data_seed $data_seed \
            --diffusion_type ode \
            --score_type score_sde \
            --attack_version rand \
            --eot_iter 20 \

[weilinie]

Hi @thuwzy, your script looks correct to me. Can you also try the standard version of Attack Linf with eps=8/255? Because the reported robust Acc is the lower value of these two settings.

[thuwzy]

Hi @weilinie , I tried standard version of Attack Linf with eps=8/255, with --fix-rand command. I got 51.56%. May be there should be some specification on the setting of odeint?

[weilinie]

@thuwzy Just wanted to check if you can successfully reproduce the ODE results?

[thuwzy]

@weilinie No, I cannot reproduce the ODE results. It seems that the performance of ODE defense is better than the reported results of ODE defense in the paper.

[weilinie]

OK, let me check what configurations we used to report the ODE results and get back here.

[thuwzy]

Thanks！

[weilinie]

@thuwzy Thanks for your feedback. I added the run scripts for getting the ODE results: run_cifar_rand_inf_ode.sh and run_cifar_stand_inf_ode.sh. I think the main difference from your previous configuration is to make sure --step_size 1e-3. Let me know if they work.

[thuwzy]

Thanks! I will try.

[thuwzy]

I run your new scripts and got the reported results same as the paper. Thanks!

[weilinie]

Nice! That's very good to know.