[New Model] <Peng2023Robust>

[ShengYun-Peng]

Paper Information

• Paper Title: Robust Principles: Architectural Design Principles for Adversarially Robust CNNs
• Paper URL: https://arxiv.org/abs/2308.16258
• Paper authors: ShengYun Peng, Weilin Xu, Cory Cornelius, Matthew Hull, Kevin Li, Rahul Duggal, Mansi Phute, Jason Martin, and Duen Horng Chau
• Venue: BMVC'23

Leaderboard Claim(s)

Add here the claim for your model(s). Copy and paste the following subsection for the number of models you want to add.

Model 1

• Architecture: RaWRN-70-16
• Dataset: CIFAR-10
• Threat Model: Linf
• eps: 8/255
• Clean accuracy: 93.27%
• Robust accuracy: 71.07%
• Additional data: false (training with 50M diffusion dataset from Better Diffusion Models Further Improve Adversarial Training)
• Evaluation method: AutoAttack (AA)
• Checkpoint and code: code, model, AA eval log

Model 2

• Architecture: RaWRN-101-2
• Dataset: ImageNet
• Threat Model: Linf
• eps: 4/255
• Clean accuracy: 73.44%
• Robust accuracy: 48.94%
• Additional data: false (PGD adversarial training)
• Evaluation method: AutoAttack (AA)
• Checkpoint and code: code, model

Model Zoo:

• [x] I want to add my models to the Model Zoo (check if true)
• [x] ~I use an architecture that is included among those here or in timm.~ If not, I added the link to the architecture implementation so that it can be added.
• [x] I agree to release my model(s) under MIT license (check if true) OR under a custom license, located here: (put the custom license URL here if a custom license is needed. If no URL is specified, we assume that you are fine with MIT)

[fra31]

Hi,

thanks for the submission, we'll add the model as soon as possible.

[fra31]

This adds the CIFAR-10 model. Could you please provide a model definition for the ImageNet classifier in a similar format (with all configurations listed as here)?

[ShengYun-Peng]

Thanks @fra31! Shall I commit to add_models_2 branch?

[fra31]

Either that or you can just add here the model definition (I guess the definition of all layers is already present in your code, so just the final architecture, similar to NormalizedWideResNet(...) for CIFAR-10, would be needed), whatever it's more convenient for you.

[ShengYun-Peng]

The model is defined as slightly different from the CIFAR-10 WRN, but I'll add both models to the same file.

[ShengYun-Peng]

Hi @fra31, could you grant me access to the add_models_2 branch? I updated the code locally on my machine, but cannot push commits.

[fra31]

Can you maybe create a PR instead? I can take care of it from there.

[ShengYun-Peng]

Sure, PR created: https://github.com/RobustBench/robustbench/pull/153

[fra31]

Great, thanks!

[fra31]

Added the models with https://github.com/RobustBench/robustbench/pull/154. Is this preprocessing for ImageNet the right one?

[ShengYun-Peng]

Right, my test_transform is the same as Res256Crop224.

[fra31]

Just another question: are the reported clean and robust accuracy for ImageNet on the entire validation set (50k points) rather than on the subset (5k points) we use for the leaderboard?

[ShengYun-Peng]

Only the 5k subset is used for AutoAttack on ImageNet. The configs are here and loaded here. The ImageNet was tested months ago, so I couldn't find the log. Sorry if that causes the confusion above.

[fra31]

It's just because for our 5k points subset I get 73.10% clean accuracy, while 73.448% for the full validation, which is much closer to (matches, depending on how one rounds) what you report. It might be also a difference in some package version though.

[ShengYun-Peng]

That makes sense.

It's just because for our 5k points subset I get 73.10% clean accuracy, while 73.448% for the full validation, which is much closer to (matches, depending on how one rounds) what you report. It might be also a difference in some package version though.

[fra31]

Do you mean that the clean accuracy is reported for the full validation set, while robust accuracy for the subset?

[ShengYun-Peng]

Oh, I meant the package differences. As shown here, I'm directly using the benchmark API from the robustbench, so there's no way the clean accuracy is on the full validation set, while the adversarial is only 5k.

[fra31]

The leaderboards should be updated.

[ShengYun-Peng]

Thanks so much! @fra31