search

HKU-TASR / Imperio

[IJCAI 2024] Imperio is an LLM-powered backdoor attack. It allows the adversary to issue language-guided instructions to control the victim model's prediction for arbitrary targets.
https://khchow.com/Imperio/
MIT License
40 stars 3 forks source link

A confusion about defense dataset #1

Open wizaaaard opened 1 month ago

wizaaaard commented 1 month ago

Dear author, thank you for your code and creative ideas, they have left a deep impression on me. I have a question regarding your code: In core.dataset.py, in each dataset class, you split the test_dataset into defense_test_dataset and defense_val_dataset. However, you do not use defense_val_dataset anywhere, and why don't you directly use test_dataset for testing? I am looking forward to your reply. Thank you again.

hkucs-kachow commented 1 month ago

Thank you for your question. We are thrilled that Imperio has left a deep impression on you.

It was supposed to be used by defenses to learn their hyperparameters. If you focus on attacks and want more data samples for testing, you can disable the split and use the entire test dataset.

wizaaaard commented 1 month ago

Could you please clarify whether the testing metrics provided in the paper were tested on the complete test dataset or on the defense_dataset in the current code?