search

AGI-Edgerunners / LLM-Adapters

Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"
https://arxiv.org/abs/2304.01933
Apache License 2.0
1.01k stars 91 forks source link

finetune accuracy is much higher than what is in the README table #30

Open CrazyElements opened 12 months ago

CrazyElements commented 12 months ago

Great work, I use the fine-tune code that can be obtained in math_running_commands to train the pre-trained model yahma/llama-7b-hf with LoRA. And then I evaluate the fine-tuned model on SVAMP, whose accuracy is 66.2%, with about 40% improvement form the previous 47.2%.

Could you please tell me why I can get a much better result? Did you update the dataset again?

Thank you!

HZQ950419 commented 12 months ago

Hi, yes, we have updated the dataset. And We will update the table soon. Sorry for the confusion!

ToheartZhang commented 11 months ago

@HZQ950419 Hi, I wonder if the updated results are obtained by models trained on math_10k.json or another training dataset?

HZQ950419 commented 11 months ago

@HZQ950419 Hi, I wonder if the updated results are obtained by models trained on math_10k.json or another training dataset?

Hi, the updated results are obtained by models trained on math_10k.json for math reasoning.

HZQ950419 commented 11 months ago

Great work, I use the fine-tune code that can be obtained in math_running_commands to train the pre-trained model yahma/llama-7b-hf with LoRA. And then I evaluate the fine-tuned model on SVAMP, whose accuracy is 66.2%, with about 40% improvement form the previous 47.2%.

Could you please tell me why I can get a much better result? Did you update the dataset again?

Thank you!

Hi,

After carefully checking, I can't reproduce your results with llama-7b-lora. Could you give more information? The command used for training and testing, the hardware used for the experiment, and so on.

Thanks!