[New Model] <Liu2023Comprehensive>

[sunrise6513]

Paper Information

• Paper Title: A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking
• Paper URL: https://arxiv.org/abs/2302.14301
• Paper authors: Chang Liu, Yinpeng Dong, Wenzhao Xiang, Xiao Yang, Hang Su, Jun Zhu, Yuefeng Chen, Yuan He, Hui Xue, Shibao Zheng

Leaderboard Claim(s)

Add here the claim for your model(s). Copy and paste the following subsection for the number of models you want to add.

Model 1

• Architecture: ConvNext-Base
• Threat Model: Linf
• eps: 4/255
• Clean accuracy: 75.84%
• Robust accuracy: 53.98%
• Additional data: false
• Evaluation method: AutoAttack
• Checkpoint and code: checkpoint code log

Model 2

• Architecture: ConvNext-Large
• Threat Model: Linf
• eps: 4/255
• Clean accuracy: 77.18%
• Robust accuracy: 56.44%
• Additional data: false
• Evaluation method: AutoAttack
• Checkpoint and code: checkpoint code log

Model 3

• Architecture: Swin_base_patch4_window7_224
• Threat Model: Linf
• eps: 4/255
• Clean accuracy: 75.34%
• Robust accuracy: 54.16%
• Additional data: false
• Evaluation method: AutoAttack
• Checkpoint and code: checkpoint code log

Model 4

• Architecture: Swin_large_patch4_window7_224
• Threat Model: Linf
• eps: 4/255
• Clean accuracy: 78.10%
• Robust accuracy: 57.06%
• Additional data: false
• Evaluation method: AutoAttack
• Checkpoint and code: checkpoint code log

Model Zoo:

• [x] I want to add my models to the Model Zoo (check if true)
• [x] I use an architecture that is included among those here or in timm. If not, I added the link to the architecture implementation so that it can be added.
• [x] I agree to release my model(s) under MIT license (check if true) OR under a custom license, located here: (put the custom license URL here if a custom license is needed. If no URL is specified, we assume that you are fine with MIT)

[fra31]

Hi,

thanks for the submission! Could you please let me know which model definitions and image preprocessing are used?

[sunrise6513]

The model definitions and image preprocessing are shown below: mu = (0.485, 0.456, 0.406) sigma = (0.229, 0.224, 0.225) [('Liu2023Comprehensive_ConvB', { 'model': lambda: normalize_model(timm.create_model( 'convnext_base', pretrained=False), mu, sigma), 'gdrive_id': None, 'preprocessing': 'Res256Crop224' }), ('Liu2023Comprehensive_ConvL', { 'model': lambda: normalize_model(timm.create_model( 'convnext_large', pretrained=False), mu, sigma), 'gdrive_id': None, 'preprocessing': 'Res256Crop224' }), ('Liu2023Comprehensive_SwinB', { 'model': lambda: normalize_model(timm.create_model( 'swin_base_patch4_window7_224', pretrained=False), mu, sigma), 'gdrive_id': None, 'preprocessing': 'Res256Crop224' }), ('Liu2023Comprehensive_SwinL', { 'model': lambda: normalize_model(timm.create_model( 'swin_large_patch4_window7_224', pretrained=False), mu, sigma), 'gdrive_id': None, 'preprocessing': 'Res256Crop224' })]

[fra31]

Adding the models I noticed that using bicubic interpolation for preprocessing instead of bilinear (default value) yields slightly higher clean accuracy (0.5-0.8%). Is it possible that it was used for training?

[sunrise6513]

Sorry, the preprocessing should be BicubicRes256Crop224 for all models.

[sunrise6513]

May I ask when will the final result be updated to the leaderboard?

[fra31]

I had to rerun the evaluation with the different preprocessing, I'll update the leaderboard in the next couple of days.

[sunrise6513]

Ok, thanks for your effort.

[fra31]

Added the models with https://github.com/RobustBench/robustbench/pull/136, please let me know if there's anything to change.

[sunrise6513]

No other changes, thanks. The only change is modifying preprocessing from Res256Crop224 to BicubicRes256Crop224.