Issues with running code

[yewang]

I'm trying to reproduce your results from your paper, but I'm having some issues when trying to run the code.

When I try to run:

python transfer_attack.py llm=llama-2 task=AlpacaEval attack=autodan defense=smoothllm

I get the error message:

FileNotFoundError: [Errno 2] No such file or directory: 'data/AlpacaEval/AlpacaEval.jsonl'

The file config/task/AlpacaEval.yaml specifies attack_log_file: data/AlpavaEval/AlpacaEval.jsonl, but this jsonl file does not exist. Instead, data/AlpacaEval/AlpacaEval-125.jsonl is provided in the repo. However, changing that configuration file seems to not fix it, since the code actually specifies data/AlpavaEval/AlpacaEval.jsonl again at line 203 of tasks.py. Changing it there seems to fix the issue, but the code still winds up crashing due to another issue:

    raise MissingMandatoryValue("Missing mandatory value: $FULL_KEY")
omegaconf.errors.MissingMandatoryValue: Missing mandatory value: smoothllm_perturbations
    full_key: smoothllm_perturbations
    object_type=dict

I also noticed some other issues:

• config/task/InstructionFollow.yaml specifies _target_: AlpacaEval, which seems incorrect.
• Line 20 of the README.md seems to have a typo in specifying defense=${ATTACK} and is missing the attack parameter. I guess this part should be attack=${ATTACK} defense=${DEFENSE}, like in line 25.

I was able to get things to start running with the incontext defense (with the following command), but then it immediately drops into the interactive python debugger, and crashes if I try to continue the script.

python transfer_attack.py llm=llama-2 task=AlpacaEval attack=autodan defense=incontext

[Question406]

hi, sorry for the late response. I've been very busy recently and didn't notice the issue, I will check the code later.

[Question406]

hi, we updated the repo, which should have resolved the error. By the way, when calling smoothllm defense, the arguments must specify perturbation type via smoothllm_perturbations=RandomSwap/RandomInsert/RandomPatch, detailed in config/defense/smoothllm.yaml.