Closed lunaryan closed 1 year ago
THUYimingLi
commented
1 year ago The script "test_SleeperAgent.py" with cifar10 dataset achieves only 9.99% ASR after poisoning:
Epoch 100[2023-06-01_00:45:58] train_acc: 99.80, test_acc: 93.11, source_asr: 8.60, full_asr: 9.99
Thank you for your report! I have contacted the developer of this method. He will check it out during this weekend.
GuanhaoGan
commented
1 year ago The script "test_SleeperAgent.py" with cifar10 dataset achieves only 9.99% ASR after poisoning:
Epoch 100[2023-06-01_00:45:58] train_acc: 99.80, test_acc: 93.11, source_asr: 8.60, full_asr: 9.99
Thank you for reporting this. We first (pre-)train a clean model and then craft poison samples and then poison the model via retraining. We are not sure whether the results you reported are from the pretraining or poisoning phase, can you provide more details on that? We apologize for the inconvenience it causes you. Also, we met some unexpected issues reproducing the reported results. We are still looking into the solution to the problem, it may take longer to solve.
lunaryan
commented
1 year ago Thanks so much for your reply! The code and full log are available at https://pastebin.com/cbrqvFDR. Please let me know if you cannot access it.
The machine is Ubuntu 18.04.6 LTS Linux 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux and enviroment is created by:
2021 conda create python=3.8 --name py3.8 2022 conda activate py3.8 2023 pip install -r requirements.txt 2024 python test_SleeperAgent.py > log
and detailed as:
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
ca-certificates 2023.05.30 h06a4308_0
certifi 2023.5.7 pypi_0 pypi
charset-normalizer 2.0.12 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
easydict 1.9 pypi_0 pypi
fonttools 4.39.4 pypi_0 pypi
idna 3.4 pypi_0 pypi
imageio 2.16.0 pypi_0 pypi
kiwisolver 1.4.4 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
lpips 0.1.4 pypi_0 pypi
matplotlib 3.5.1 pypi_0 pypi
ncurses 6.4 h6a678d5_0
numpy 1.22.4 pypi_0 pypi
opencv-python 4.7.0.72 pypi_0 pypi
openssl 1.1.1t h7f8727e_0
packaging 23.1 pypi_0 pypi
pandas 2.0.2 pypi_0 pypi
pillow 9.5.0 pypi_0 pypi
pip 23.0.1 py38h06a4308_0
pyparsing 3.0.9 pypi_0 pypi
python 3.8.16 h7a1cb2a_3
python-dateutil 2.8.2 pypi_0 pypi
pytz 2023.3 pypi_0 pypi
readline 8.2 h5eee18b_0
requests 2.26.0 pypi_0 pypi
scipy 1.7.3 pypi_0 pypi
seaborn 0.11.2 pypi_0 pypi
setuptools 67.8.0 py38h06a4308_0
six 1.16.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
termcolor 1.1.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
torch 1.8.0+cu111 pypi_0 pypi
torchaudio 0.8.0 pypi_0 pypi
torchvision 0.9.0+cu111 pypi_0 pypi
tqdm 4.62.3 pypi_0 pypi
typing-extensions 4.6.3 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 1.26.16 pypi_0 pypi
wheel 0.38.4 py38h06a4308_0
xz 5.4.2 h5eee18b_0
zlib 1.2.13 h5eee18b_0 Thanks again!
GuanhaoGan
commented
1 year ago Thanks so much for your reply! The code and full log are available at https://pastebin.com/cbrqvFDR. Please let me know if you cannot access it.
The machine is Ubuntu 18.04.6 LTS Linux 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux and enviroment is created by:
2021 conda create python=3.8 --name py3.8 2022 conda activate py3.8 2023 pip install -r requirements.txt 2024 python test_SleeperAgent.py > log
and detailed as:
_libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu ca-certificates 2023.05.30 h06a4308_0 certifi 2023.5.7 pypi_0 pypi charset-normalizer 2.0.12 pypi_0 pypi cycler 0.11.0 pypi_0 pypi easydict 1.9 pypi_0 pypi fonttools 4.39.4 pypi_0 pypi idna 3.4 pypi_0 pypi imageio 2.16.0 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 libffi 3.4.4 h6a678d5_0 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 lpips 0.1.4 pypi_0 pypi matplotlib 3.5.1 pypi_0 pypi ncurses 6.4 h6a678d5_0 numpy 1.22.4 pypi_0 pypi opencv-python 4.7.0.72 pypi_0 pypi openssl 1.1.1t h7f8727e_0 packaging 23.1 pypi_0 pypi pandas 2.0.2 pypi_0 pypi pillow 9.5.0 pypi_0 pypi pip 23.0.1 py38h06a4308_0 pyparsing 3.0.9 pypi_0 pypi python 3.8.16 h7a1cb2a_3 python-dateutil 2.8.2 pypi_0 pypi pytz 2023.3 pypi_0 pypi readline 8.2 h5eee18b_0 requests 2.26.0 pypi_0 pypi scipy 1.7.3 pypi_0 pypi seaborn 0.11.2 pypi_0 pypi setuptools 67.8.0 py38h06a4308_0 six 1.16.0 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 termcolor 1.1.0 pypi_0 pypi tk 8.6.12 h1ccaba5_0 torch 1.8.0+cu111 pypi_0 pypi torchaudio 0.8.0 pypi_0 pypi torchvision 0.9.0+cu111 pypi_0 pypi tqdm 4.62.3 pypi_0 pypi typing-extensions 4.6.3 pypi_0 pypi tzdata 2023.3 pypi_0 pypi urllib3 1.26.16 pypi_0 pypi wheel 0.38.4 py38h06a4308_0 xz 5.4.2 h5eee18b_0 zlib 1.2.13 h5eee18b_0Thanks again!
Thank you for your detailed reply! I will look into this problem shortly.
GuanhaoGan
commented
1 year ago Sorry for the late reply. I'm occupied with other stuff recently and the problem is rather difficult. I try to reproduce your results on 3 different servers, but all experiments went well with at least 50% ASR. I also compare our code with the official code which is complexly encapsulated, but cannot find any significant difference. So I guess the problem arises from the experiment settings. By setting source_num to 50 or 1000 for CIFAR-10 when calling core.SleeperAgent, I can achieve better ASR (~90%). You can try this setting and see if this helps. B.T.W, the ASR metric here denotes the average rate at which patched source images are successfully classified as the target class (See Appendix B.1 in the original paper), which is termed source_asr in our implementation.
lunaryan
commented
1 year ago Thanks so much for your reply! I will try your suggested setting.
The script "test_SleeperAgent.py" with cifar10 dataset achieves only 9.99% ASR after poisoning:
Epoch 100[2023-06-01_00:45:58] train_acc: 99.80, test_acc: 93.11, source_asr: 8.60, full_asr: 9.99