search

AGI-Edgerunners / LLM-Adapters

Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"
https://arxiv.org/abs/2304.01933
Apache License 2.0
1.08k stars 103 forks source link

finetune accuracy is much higher than what is in the README table #30

Open CrazyElements opened 1 year ago

CrazyElements commented 1 year ago

Great work, I use the fine-tune code that can be obtained in math_running_commands to train the pre-trained model yahma/llama-7b-hf with LoRA. And then I evaluate the fine-tuned model on SVAMP, whose accuracy is 66.2%, with about 40% improvement form the previous 47.2%.

Could you please tell me why I can get a much better result? Did you update the dataset again?

Thank you!

HZQ950419 commented 1 year ago

Hi, yes, we have updated the dataset. And We will update the table soon. Sorry for the confusion!

ToheartZhang commented 1 year ago

@HZQ950419 Hi, I wonder if the updated results are obtained by models trained on math_10k.json or another training dataset?

HZQ950419 commented 1 year ago

@HZQ950419 Hi, I wonder if the updated results are obtained by models trained on math_10k.json or another training dataset?

Hi, the updated results are obtained by models trained on math_10k.json for math reasoning.

HZQ950419 commented 1 year ago

Great work, I use the fine-tune code that can be obtained in math_running_commands to train the pre-trained model yahma/llama-7b-hf with LoRA. And then I evaluate the fine-tuned model on SVAMP, whose accuracy is 66.2%, with about 40% improvement form the previous 47.2%.

Could you please tell me why I can get a much better result? Did you update the dataset again?

Thank you!

Hi,

After carefully checking, I can't reproduce your results with llama-7b-lora. Could you give more information? The command used for training and testing, the hardware used for the experiment, and so on.

Thanks!