lhfowl
commented
3 years ago Hello,
It's hard to say exactly what's happening without knowing your training pipeline. I just did a fresh clone of the repo and ran it through our evaluation pipeline and the victim network got 7.26% accuracy on a clean CIFAR-10 testset. The validation accuracies printed every 5 epochs were: [[43.7, 24.02, 14.33, 8.1, 10.6, 8.9, 8.74, 9.47, 7.72, 6.51, 7.43, 6.37, 7.74, 6.7, 7.11, 7.05, 7.13, 7.08, 7.66, 7.59, 7.26]]
We use a fairly standard training routine for CIFAR-10 (SGD, linear scheduler, 100 epochs, standard augmentations, etc.) Can you share some more info about how you're evaluating the poisons? You could also try duplicating our hyperparameters in your setup.
Also, have you tried generating/running our targeted adversarial poisons through your setup? We note in the paper (sections 3.4, 4, A9) that generating an effective untargeted attack is noticeably more difficult than a targeted attack, and thus we suggest our class targeted attack as the more reliable and potent form of poisoning. We do find that the untargeted poison success degrades significantly on ImageNet, and in general, the untargeted attacks are more unstable to initialization. It would certainly be interesting if they are also relatively unstable to changes in the training routine (although our evaluation routine is "unknown" at the time of poisoning).
Edit: I have since cloned a very popular, completely black box (to our poisoning pipeline) CIFAR-10 training repo and loaded our poisoned dataset into their training routine. In their setup, the victim's accuracy is 8.150%.
WeiqiPeng0
Dear authors,
We trained a DNN model on the untargeted Cifar-10 dataset you provided. However, the poisoning effect seems not to be as powerful as you mentioned--the evaluation accuracy on a clean test set does not decrease too much (no more than -20% on our model).
We failed to use your code for evaluation because of same execution errors on windows. Would you advise how to reproduce the result? thank you!