Closed prateeky2806 closed 10 months ago
Hi there,
Thank you for your response. I've reviewed the information you provided, and everything seems to be in order.
You can find the evaluation guidance on this page.
Any of the checkpoints in the model zoo can be loaded with the quantization method by using this function. I noticed that the evaluation team was not involved in the quantization implementation. I plan to submit a pull request this weekend to address this.
If you have any further questions or concerns, please feel free to let me know.
I've checked the code and revisited the BBH implementation. Other evaluation codes are in the revisiting progress.
If you would like to evaluate on BBH, the running script could be like the following. In this case, it initializes a 7b llama2 model, and loads official llama2 weights as well as PEFT weights (3.35MB, alpaca_llamaPeft_normBias). Since the --quant
flag is activated, the base model will be quantized to 4bit by using QLoRA implementation, while the PEFT weights are stayed in higher precision(fp32/fp16/bp16).
cd LLaMA2-Accessory/light-eval
torchrun --nproc-per-node=1 --master_port 23456 src/eval_bbh.py \
--llama_type llama_peft \
--llama_config <path-to>/Llama-2-7b/params.json \
--tokenizer_path <path-to>/Llama-2-7b/tokenizer.model \
--pretrained_path <path-to>/Llama-2-7b/ <path-to>/alpaca_llamaPeft_normBias \
--data_dir data/BIG-Bench-Hard \
--quant
Thank you so much! I was wondering if you could provide a way to find out which checkpoints are Lora/Qlora checkpoints and which are fully finetuned models so that it is easier for people to select the correct models from the model zoo.
Thank you so much! I was wondering if you could provide a way to find out which checkpoints are Lora/Qlora checkpoints and which are fully finetuned models so that it is easier for people to select the correct models from the model zoo.
Thank you for your suggestion. In fact, we have been doing so. Parameter efficient fine-tuning checkpoints are generally labeled with "peft" or "llamaAdapter", e.g. alpacaLlava_llamaQformerv2Peft_13b and alpaca_llamaPeft_normBias, and those without special labels are full-parameter fine-tune methods
Hi, thank you for creating this amazing repository. I have two questions.
It would be great if you can help with this.
Thanks, Prateek