On adv. robust classifiers having better transfer performance

[sayakpaul]

I am referring to this paper: Do Adversarially Robust ImageNet Models Transfer Better? (Salman et al., NeurIPS'20). The authors show that adversarially robust models have better transfer learning performance. They attribute this gain to the following factors:

• Adv. robust models have better-informed gradients.
• Adv. robust models can learn representations that have better and easier invertibility.

I think it might be interesting to use nsl to train models on ImageNet-1k and verify how well these observations hold.

[DualityGap]

Interesting, thanks for the nice suggestion, Sayak! Meanwhile, if you have any results, feel free to share.

[sayakpaul]

As an individual, it's currently not possible for me to train ImageNet-1k scale models without having the required resources and guidance. This is why I just wanted to pass this suggestion to the NSL team as I think it would be a very nice use case to apply NSL on.