Does f(x') replace the label or the prediction in the regularization term?

[yaircarmon]

Thank you for sharing an implementation of TRADES - it really helps understand your paper. However, there one thing was unclear to me when comparing the paper and the code. According to the paper (and also the github readme), in the regularization term the adversarial prediction f(X’) plays the role of the label (i.e. second argument to $\mathcal{L}$), while f(X) remains in the same place as in the natural loss. In contrast, in the regularization term implemented in trades.py, model(x_natural) plays the role of the label (second argument to criterion_kl), and model(x_adv) forms the prediction.

Which version is the correct one (i.e. the one used to train the publicly available CIFAR-10 model)?

[hongyanz]

Hi Carmon,

Thanks for your interest. In the theoretical analysis of our paper, we study the two-class case of the form phi(f(X)f(X')), where there is no concept of 'prediction' and 'target' here (since there is only one argument in the function phi). For the multiple-class case, we propose using cross-entropy loss L as a heuristic surrogate of phi(f(X)f(X')). There is no standard to choose either f(X) or f(X') as the 'target'. According to our experiments, the performance of two setups (use either f(X) or f(X') as the target) does not differ too much.

Please let me know if you have further questions. Thanks.