[New Model] <Bai2023Improving>

[Bai-YT]

Paper Information

• Paper Title: Improving the Accuracy-Robustness Trade-off of Classifiers via Adaptive Smoothing.
• Paper URL: https://arxiv.org/abs/2301.12554. Note that the following results were achieved after the preprint paper was uploaded to ArXiv, and are thus not presented in the linked paper yet.
• Paper Authors: Yatong Bai, Brendon G Anderson, Aerin Kim, Somayeh Sojoudi.

Leaderboard Claim(s)

Model 1

• Architecture: ResNet-152 + WideResNet-70-16 + mixing network
• Dataset: CIFAR-100
• Threat Model: Linf
• eps: 8 / 255
• Clean accuracy: 85.21 %
• Robust accuracy: 38.72 %
• Additional data: True (via the accurate base classifier. Additionally, the robust base classifier requires 50M synthetic images.)
• Evaluation method: AutoAttack
• Checkpoint and code: The checkpoints and code are available here.

Model 2

• Architecture: ResNet-152 + WideResNet-70-16 + mixing network
• Dataset: CIFAR-100
• Threat Model: Linf
• eps: 8 / 255
• Clean accuracy: 80.18 %
• Robust accuracy: 35.15 %
• Additional data: True (via both the accurate base classifier and the robust base classifier.)
• Evaluation method: AutoAttack
• Checkpoint and code: The checkpoints and code are available here.

Model Zoo:

• [x] I want to add my models to the Model Zoo (check if true).
• [x] I use an architecture that is included among those here or in timm. If not, I added the link to the architecture implementation so that it can be added.
• [x] I agree to release my model(s) under MIT license (check if true) OR under a custom license, located here: (put the custom license URL here if a custom license is needed. If no URL is specified, we assume that you are fine with MIT).

[Bai-YT]

@fra31 Just hoped to follow up on this issue. If you need additional details, please feel free to let me know! Thank you very much.

[fra31]

Hi,

thanks for the submission! I'll have a look in the next few days.

[Bai-YT]

Thank you!

[fra31]

I added with https://github.com/RobustBench/robustbench/commit/ea17b3a1862256f75e26c3e19e44cc16ce84eff7 the models. I made small changes to integrate them in the model zoo framework, please let me know if that's fine. Also, are https://github.com/RobustBench/robustbench/blob/ea17b3a1862256f75e26c3e19e44cc16ce84eff7/robustbench/utils.py#L209 and following lines (creating the new dictionary) necessary, since the checkpoint already contains a "bn" key? When I removed them the accuracy of the models doesn't change.

[Bai-YT]

Thank you for your time! Confirm that lines 209 to 213 can be removed.

[Bai-YT]

Hi @fra31, just hoped to follow up about when we can expect the numbers to be added to the RobustBench website. Thanks a lot for your time and effort!

[fra31]

Sorry for the delay on this, I'll try to complete it soon.

[fra31]

Added with https://github.com/RobustBench/robustbench/pull/142, if it looks fine to you, I'll merge it and update the leaderboard.

[Bai-YT]

Looks great! Thanks a lot for your time!

[Bai-YT]

Hello @fra31, I hope you are doing great! Thank you for listing our method on RobustBench.

We would like to add a CIFAR-10 model to our results. The claims are the following:

Model 3

• Architecture: ResNet-152 + WideResNet-70-16 + mixing network
• Dataset: CIFAR-10
• Threat Model: Linf
• eps: 8 / 255
• Clean accuracy: 95.03 % 95.23 %
• Robust accuracy: 67.97 % 68.06 %
• Additional data: True. The accurate base classifier was pre-trained on ImageNet-21k. Additionally, the robust base classifier used 50M synthetic images.
• Evaluation method: AutoAttack
• Checkpoint and code: The checkpoints and code are available here (please see the edits made on Aug 3, 2023).

We greatly appreciate your time and effort. If you need additional details, please feel free to let us know.

[fra31]

Hi,

I'll add the new model as soon as possible.

[fra31]

It seems that the link to download the standard classifier still points to the model fine-tuned on CIFAR-100. Could you please provide the one for CIFAR-10?

[Bai-YT]

Really sorry for the issue. Here is the link. I have also updated the link in my repo. Thanks for your time!

[fra31]

Thanks, it works fine now. It seems from here that the normalization for CIFAR-100 is used even for CIFAR-10, just wanted to check that this is the intended behavior (the difference in clean accuracy is around 0.2% from what I've seen).

[Bai-YT]

Thank you for catching this. We forgot to change the normalization settings. Changing lines 51 to 62 to the following should fix the issue:

    mean = dm_rn.CIFAR100_MEAN if num_classes == 100 else dm_rn.CIFAR10_MEAN
    std = dm_rn.CIFAR100_STD if num_classes == 100 else dm_rn.CIFAR10_STD
    if rob_model_type == 'wrn7016':
        # Use DeepMind's Swish
        rob_model = dm_rn.WideResNet(
            num_classes=num_classes, activation_fn=dm_rn.Swish,
            depth=70, width=16, mean=mean, std=std
        )
    elif rob_model_type == 'wrn7016_silu':
        # Use PyTorch's SiLU
        rob_model = dm_rn.WideResNet(
            num_classes=num_classes, activation_fn=nn.SiLU, 
            depth=70, width=16, mean=mean, std=std
        )

We need to rerun the evaluation on our side and may need to slightly modify our claim. Will post the modified claim shortly. Sorry for the inconvenience.

[Bai-YT]

The new claim is:

• Clean accuracy: 95.23 %
• Robust accuracy: 68.06 %

Sorry for the wait!

[fra31]

Sorry for the delay, just added the model with https://github.com/RobustBench/robustbench/pull/149, let me know if it looks fine.

[Bai-YT]

Thank you very much for your time, and it looks good to me. I look forward to it being added to the leaderboard. Please have a wonderful day.

[Bai-YT]

Hi! Just wanted to update you that this paper has been accepted to the SIAM Journal on Mathematics of Data Science (SIMODS). Would you mind updating the leaderboard? Thank you.