运行test_IBD-PSC.py遇到的一个问题

[aosodofogo]

首先是运行test_BadNets.py得到中毒的预训练模型（ResNet-18_CIFAR-10_BadNets_2024-08-19/ckpt_epoch_170.pth） 结果如下： ==========Test result on benign test dataset========== [2024-08-19_23:07:20] Top-1 correct / Total: 8786/10000, Top-1 accuracy: 0.8786, Top-5 correct / Total: 9926/10000, Top-5 accuracy: 0.9926, mean loss: 0.449150949716568, time: 5.905892848968506

==========Test result on poisoned test dataset========== [2024-08-19_23:07:23] Top-1 correct / Total: 9746/10000, Top-1 accuracy: 0.9746, Top-5 correct / Total: 9962/10000, Top-5 accuracy: 0.9962, mean loss: 0.16978201270103455, time: 8.407617807388306

然后运行test_IBD-PSC.py，在

========== ResNet-18_CIFAR-10_Attack_Defense IBD-PSC ==========

下面修改加载的预训练模型，

但是当我debug到 defense = core.IBD_PSC(model=badnet_model, valset=val_set) print(f'the BA and ASR of the original BadNets model: ............. ') test(model_name, dataset_name, attack_name, defense_name, testset, poisoned_testset, defense, None)的时候

测试得到的结果如下： ==========Test result on BA========== [2024-08-19_23:13:23] Top-1 correct / Total: 4861/10000, Top-1 accuracy: 0.4861, Top-5 correct / Total: 7444/10000, Top-5 accuracy: 0.7444, time: 7.864623546600342

==========Test result on ASR_NoTarget========== [2024-08-19_23:13:28] Top-1 correct / Total: 5507/10000, Top-1 accuracy: 0.5507, Top-5 correct / Total: 7475/10000, Top-5 accuracy: 0.7475, time: 4.886025428771973

请问您有什么建议吗，是不是哪一步是我做错了

[LandAndLand]

首先是运行test_BadNets.py得到中毒的预训练模型（ResNet-18_CIFAR-10_BadNets_2024-08-19/ckpt_epoch_170.pth） 结果如下： ==========Test result on benign test dataset========== [2024-08-19_23:07:20] Top-1 correct / Total: 8786/10000, Top-1 accuracy: 0.8786, Top-5 correct / Total: 9926/10000, Top-5 accuracy: 0.9926, mean loss: 0.449150949716568, time: 5.905892848968506

==========Test result on poisoned test dataset========== [2024-08-19_23:07:23] Top-1 correct / Total: 9746/10000, Top-1 accuracy: 0.9746, Top-5 correct / Total: 9962/10000, Top-5 accuracy: 0.9962, mean loss: 0.16978201270103455, time: 8.407617807388306

然后运行test_IBD-PSC.py，在

========== ResNet-18_CIFAR-10_Attack_Defense IBD-PSC ==========

下面修改加载的预训练模型，

但是当我debug到 defense = core.IBD_PSC(model=badnet_model, valset=val_set) print(f'the BA and ASR of the original BadNets model: ............. ') test(model_name, dataset_name, attack_name, defense_name, testset, poisoned_testset, defense, None)的时候

测试得到的结果如下： ==========Test result on BA========== [2024-08-19_23:13:23] Top-1 correct / Total: 4861/10000, Top-1 accuracy: 0.4861, Top-5 correct / Total: 7444/10000, Top-5 accuracy: 0.7444, time: 7.864623546600342

==========Test result on ASR_NoTarget========== [2024-08-19_23:13:28] Top-1 correct / Total: 5507/10000, Top-1 accuracy: 0.5507, Top-5 correct / Total: 7475/10000, Top-5 accuracy: 0.7475, time: 4.886025428771973

请问您有什么建议吗，是不是哪一步是我做错了

你好，感谢对我们工作的关注！影响攻击效果的因素主要有trigger pattern，target label。请你检查一下训练和测试BadNets model所使用的trigger pattern和target label是否统一（源代码中，test_BadNets.py的target label设置的应该是1，而test_IBD-PSC.py是0）。在IBD-PSC.py文件中，测试BA和ASR的代码和训练BadNets模型期间测试代码是一致的，按理来说如果训练的没问题，不会出现BA比较低的情况。训练epoch应该不是170，建议你加载最后一个epoch得到的中毒模型来测试。祝实验顺利

[aosodofogo]

感谢您的回复！我训练了更多epoch（设置为600）并修改target label，但测试BA的准确率依然很低（测试BA的数据是clean的，其实和中毒设置没关系）： ==========Test result on BA========== [2024-08-20_16:50:21] Top-1 correct / Total: 4169/10000, Top-1 accuracy: 0.4169, Top-5 correct / Total: 7403/10000, Top-5 accuracy: 0.7403, time: 9.512231588363647

我做了以下检查，确认是否是model或者dataset的问题： 首先尝试不加载模型，得到得BA结果如下： ==========Test result on BA========== [2024-08-20_16:59:29] Top-1 correct / Total: 965/10000, Top-1 accuracy: 0.0965, Top-5 correct / Total: 4942/10000, Top-5 accuracy: 0.4942, time: 8.94313383102417 这个结果表示没有预训练模型的时候。10分类的准确率约为0.1。和之前相比证明预训练模型发挥了作用，但达不到理想的效果

然后我在test_IBD-PSC.py中测试badnets.test(schedule)的结果（加载预训练模型），得到结果如下： ==========Test result on benign test dataset========== [2024-08-20_17:02:23] Top-1 correct / Total: 8779/10000, Top-1 accuracy: 0.8779, Top-5 correct / Total: 9827/10000, Top-5 accuracy: 0.9827, mean loss: 0.182246470451355, time: 2.7533140182495117 ==========Test result on poisoned test dataset========== [2024-08-20_17:02:27] Top-1 correct / Total: 9754/10000, Top-1 accuracy: 0.9754, Top-5 correct / Total: 9930/10000, Top-5 accuracy: 0.993, mean loss: 0.093202605843544, time: 3.9478306770324707 这个结果是正常的，预训练模型在badnets.test()中是有用的，但在BA测试中有用，但没有达到badnets.test的效果。

然后检查testset，似乎也没问题。请问您有更好的建议吗

[LandAndLand]

然后我在test_IBD-PSC.py中测试badnets.test(schedule)的结果（加载预训练模型），得到结果如下： ==========Test result on benign test dataset========== [2024-08-20_17:02:23] Top-1 correct / Total: 8779/10000, Top-1 accuracy: 0.8779, Top-5 correct / Total: 9827/10000, Top-5 accuracy: 0.9827, mean loss: 0.182246470451355, time: 2.7533140182495117 ==========Test result on poisoned test dataset========== [2024-08-20_17:02:27] Top-1 correct / Total: 9754/10000, Top-1 accuracy: 0.9754, Top-5 correct / Total: 9930/10000, Top-5 accuracy: 0.993, mean loss: 0.093202605843544, time: 3.9478306770324707

抱歉我不太明白：“然后我在test_IBD-PSC.py中测试badnets.test(schedule)的结果（加载预训练模型），得到结果如下： ==========Test result on benign test dataset========== [2024-08-20_17:02:23] Top-1 correct / Total: 8779/10000, Top-1 accuracy: 0.8779, Top-5 correct / Total: 9827/10000, Top-5 accuracy: 0.9827, mean loss: 0.182246470451355, time: 2.7533140182495117 ==========Test result on poisoned test dataset========== [2024-08-20_17:02:27] Top-1 correct / Total: 9754/10000, Top-1 accuracy: 0.9754, Top-5 correct / Total: 9930/10000, Top-5 accuracy: 0.993, mean loss: 0.093202605843544, time: 3.9478306770324707“ 这里似乎说明BA为85%以上，结果看起来是正常的

[aosodofogo]

解决问题了！ 我把代码# if GPU_num > 1: model = nn.DataParallel(model, device_ids=device_ids, output_device=device_ids[0])注释掉之后可以得到正常的BA测试结果了，之前BA只有大约一半的准确率