cannot reproduce the MMLU accuracy claimed in paper, could you release the script?

yuhuixu1993 / qa-lora

Official PyTorch implementation of QA-LoRA

MIT License

113 stars 10 forks source link

cannot reproduce the MMLU accuracy claimed in paper, could you release the script? #28

Open wenjingk-xilinx opened 6 months ago

wenjingk-xilinx commented 6 months ago

Hi, I try to reproduce the llama7b finetune on MMLU, I get the maximum 5-shot eval accuracy is 36.5% with 4bits, 32.8% with 3bits. Could you please kindly specify your training configuration or release the training script？

xxw11 commented 6 months ago

Hi,using the default parameter settings in qalora.py can replicate the results of the paper. For details on the evaluation, you can refer to the paper.