search

dvlab-research / LBGAT

Learnable Boundary Guided Adversarial Training (ICCV2021)
https://arxiv.org/abs/2011.11164
MIT License
33 stars 2 forks source link

Choice of the student and teacher network #9

Closed futakw closed 10 months ago

futakw commented 11 months ago

Hi, thanks for your great work!

I am curious about how to choose the teacher network.

On the CIFAR10 dataset, when I tried {student=ResNet18_cifar, teacher=ResNet18_cifar}, the natural and robust accuracy (against PGD-20) is 72% and 48%, which is quite bad. Therefore, I conjectured that the relationship between student and teacher is important.

Is there any ablation study regarding the choice of teacher networks? Or is there any hyper-parameter setting to make the student of ResNet18_cifar work?

jiequancui commented 11 months ago

Hi,

Thanks for your interest in our work.

LBGAT requires a large model capacity, like WideResNet34-10, and WideResNet34-20. To achieve better results on small networks like ResNet-18 with LBGAT, you might need to tune some hyper-parameters, like weight-decay or learning rate. Moreover, alpha is a trade-off hyper-parameter for natural accuracy and robust accuracy. A larger alpha can achieve stronger robustness at the cost of some natural accuracy.

Best,

futakw commented 10 months ago

Hi, thank you so much for your quick answer. I appreciate it.

I have one more related question.

When using large enough networks, such as WideResNet34-10, does the size of the teacher matter? By default, you use ResNet18 for the teacher, but is it based on some analysis or some idea?

jiequancui commented 10 months ago

Hi,

We have tried the following two settings.

  1. teacher : WideResNet34-10, student: WideResNet34-10
  2. teacher: ResNet18, student: WideResNet34-10.

Similar results are observed with the two settings.

Thanks.

futakw commented 10 months ago

Hi, Thank you so much for your quick reply :)

P.S. There seemed to be something wrong with my code (which I'm not sure of now), and ResNet18 on CIFAR10 achieved 85.50% and 49.26% of Clean accuracy and AutoAttack accuracy.