search

MiracleHH / CBA

Composite Backdoor Attacks Against Large Language Models
9 stars 2 forks source link

GPU Run Time #1

Closed SteveZ-Cal closed 7 months ago

SteveZ-Cal commented 7 months ago

  1. How long does it took if I just run the backdoor attack on the Emotion dataset with the pre-trained LLaMA-7B model and 10% poisoning ratio using (4 A100 40GB GPU) as listed in Attacks in NLP tasks. Can you also give some rough estimates to other section on how long it will took (ex: evaluation)?

  2. Is it reasonable to just run on one Tesla T4 due to resource constraint? If so, how long will it took?

  3. If I want to run with different number or type of gpu, do I need to change the input command? If so, how?

  4. I received

OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like huggyllama/llama-7b is not the path to a directory containing a file named config.json.

when I try to run the backdoor attack on the Emotion dataset with the pre-trained LLaMA-7B model and 10% poisoning ratio. It seems the Meta requires you to sign to get access the mode. Just to confirm, do we need to create a directory huggyllama/llama-7b with llama-7b model or its params.json under nlp directory? Also, it seems that llama-7b no longer available on the hugging face, if I use llama2-7b is there a need to change the input command?

MiracleHH commented 7 months ago

Hi @SteveZ-Cal, thanks for your interest in our work! Below are the detailed responses to your questions.

  1. We have provided the rough time estimates for our experiments in Appendix B in our paper. Specifically, it took about 1 hour to train the backdoored model and 3 minutes to give inference-phase predictions on 1,000 queries for the experiment you mentioned.

  2. It is possible to run the experiments on GPUs with limited resources like Tesla T4. Our experiments in NLP tasks are mainly based on the QLoRA framework. According to its blog, training LLaMA 7B and 13B models on an NVIDIA T4 (16GB) with appropriate settings (e.g., a small batch size of 1) is feasible. However, it might be very time-consuming to run the experiments. I have not run the experiments on Tesla T4, but I guess it would take more than 10 hours to run the experiment with only one Tesla T4 GPU and a batch size of 1 (you can try a larger batch size to speed up this process as long as it does not lead to an OOM error).

  3. According to the number of GPUs you are using, you can change the parameter --nproc_per_node in the input command to run the experiments.

  4. To use LLaMA and LLaMA2 models, you should first create an access token with your Hugging Face account, and then fill and submit the forms (form 1 and form 2) to Meta to apply for model access. Note that, the email address you use to apply for model access should be the same as that of your Hugging Face account. After you have been granted access, you can replace Line 376 and Line 426 with token='YOUR_ACCESS_YOKEN' to run the experiments.

There is no need to manually create an additional directory for each model. When you run the experiments with the LLaMA2-7B model, set the parameter --model_name_or_path in the input command as meta-llama/Llama-2-7b-hf.

MiracleHH commented 7 months ago

@SteveZ-Cal Moreover, when you run the experiments with a new model, you should also change the --output_dir parameter accordingly in the input command to specify a new file directory to store the model weights, which will be subsequently used for evaluation (i.e., the --adapter_path parameter).