Closed ZhouZhanJi closed 9 months ago
Hello, the sft_gsm8k.yaml
file should be the file you used to fine-tune. May I ask which yaml file you are using for fine tuning?
Hello, the
sft_gsm8k.yaml
file should be the file you used to fine-tune. May I ask which yaml file you are using for fine tuning? Hello. I used the file named 'llama_modelscope.yaml' to fine-tune, and the dataset file is alpaca_data.json in Stanford Alpaca. And then I want to use the file federatedscope/llm/eval/eval_for_gsm8k/eval.py to evaluate the model. Does the dataset for fine-tuning the model and the dataset for evaluation need to be the same?
When you use eval_for_gsm8k/eval.py
to evaluate your LLM, it already contains a loading of the evaluation dataset, which is "gsm8k_test.jsonl", thus the usage of the provided yaml file is to provide the model information. In this case, you should run the code python federatedscope/llm/eval/eval_for_gsm8k/eval.py --cfg federatedscope/llm/baseline/llama_modelscope.yaml
. This will load the ckpt whose path is exactly the federate.save_to
term in llama_modelscope.yaml
.
As for the training dataset and evaluation dataset, of course, they should not be the same, but may be related. For more details, you can refer to our paper FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning. :)
When you use
eval_for_gsm8k/eval.py
to evaluate your LLM, it already contains a loading of the evaluation dataset, which is "gsm8k_test.jsonl", thus the usage of the provided yaml file is to provide the model information. In this case, you should run the codepython federatedscope/llm/eval/eval_for_gsm8k/eval.py --cfg federatedscope/llm/baseline/llama_modelscope.yaml
. This will load the ckpt whose path is exactly thefederate.save_to
term inllama_modelscope.yaml
. As for the training dataset and evaluation dataset, of course, they should not be the same, but may be related. For more details, you can refer to our paper FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning. :)
Ok, it's very nice for you to answer my question, I will try it again by following you advice. Thank you!
I am a newbie in large language model federated learning, and now I had finished the fine-tuned part, and I have a problem now, when I run "
python federatedscope/llm/eval/eval_for_gsm8k/eval.py --cfg sft_gsm8k.yaml
in theeval_for_gsm8k
directory", the problem "FileNotFoundError: [Errno 2] No such file or directory: 'sft_gsm8k.yaml'" appeared. Is this a missing file? Where can I find the file sft_gsm8k.yaml? Looking forward to getting your reply