search

nmndeep / revisiting-at

[NeurIPS 2023] Code for the paper "Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models"
37 stars 3 forks source link

RobustAcc for L∞=8/255 models. #5

Closed OUTOFTEN closed 9 months ago

OUTOFTEN commented 9 months ago

Could authors publish the RobustAcc about L∞=8/255 imagenet models? I tried to test the robustAcc of the models. But they are very low.

nmndeep commented 9 months ago

Hi, for after 8/255 finetuning the model -- evaluation with AA: ConvNext-B-CvSt: 71.7% clean acc. with 33.2% robust acc. ConvNext-T-Cvst: 68.6% clean acc. with 29.5% robust acc.

Can you match these numbers?

OUTOFTEN commented 9 months ago

Hi, for after 8/255 finetuning the model -- evaluation with AA: ConvNext-B-CvSt: 71.7% clean acc. with 33.2% robust acc. ConvNext-T-Cvst: 68.6% clean acc. with 29.5% robust acc.

Can you match these numbers?

Thanks for your reply, I have matched the numbers. I checked my code and found I did a normalization for images by std=[0.229, 0.224, 0.225] mean=[0.485, 0.456, 0.406]. The normalization often be used for normally trained models.

I don't have any experience with training models. I am an attacker to attack the models. And in my research, I find that data normalization is very important for evaluating attack performance.

So, I want to know, is data normalization unnecessary for building robust models?

Waiting for your reply. Thanks again.

nmndeep commented 9 months ago

Hi, As we have trained the models for significant epochs - not having normalization (even though the pre-trained clean models were trained with normalization) does not seem to have any effect. We did this since our evaluations are with AutoAttack (AA) which expects the images in [0,1] i.e., non-normalized. One can normalize while training and test with AA but then one needs to prepend a normalization layer to the model itself.

Hope this helps.

OUTOFTEN commented 9 months ago

Hi, As we have trained the models for significant epochs - not having normalization (even though the pre-trained clean models were trained with normalization) does not seem to have any effect. We did this since our evaluations are with AutoAttack (AA) which expects the images in [0,1] i.e., non-normalized. One can normalize while training and test with AA but then one needs to prepend a normalization layer to the model itself.

Hope this helps.

This is helpful for me, thanks