can't find sft_gsm8k.yaml in federatedscope/llm/eval/eval_for_gsm8k

[ZhouZhanJi]

I am a newbie in large language model federated learning, and now I had finished the fine-tuned part, and I have a problem now, when I run "python federatedscope/llm/eval/eval_for_gsm8k/eval.py --cfg sft_gsm8k.yaml in the eval_for_gsm8k directory", the problem "FileNotFoundError: [Errno 2] No such file or directory: 'sft_gsm8k.yaml'" appeared. Is this a missing file? Where can I find the file sft_gsm8k.yaml? Looking forward to getting your reply

[qbc2016]

Hello, the sft_gsm8k.yaml file should be the file you used to fine-tune. May I ask which yaml file you are using for fine tuning?

[ZhouZhanJi]

Hello, the sft_gsm8k.yaml file should be the file you used to fine-tune. May I ask which yaml file you are using for fine tuning? Hello. I used the file named 'llama_modelscope.yaml' to fine-tune, and the dataset file is alpaca_data.json in Stanford Alpaca. And then I want to use the file federatedscope/llm/eval/eval_for_gsm8k/eval.py to evaluate the model. Does the dataset for fine-tuning the model and the dataset for evaluation need to be the same?

[qbc2016]

When you use eval_for_gsm8k/eval.py to evaluate your LLM, it already contains a loading of the evaluation dataset, which is "gsm8k_test.jsonl", thus the usage of the provided yaml file is to provide the model information. In this case, you should run the code python federatedscope/llm/eval/eval_for_gsm8k/eval.py --cfg federatedscope/llm/baseline/llama_modelscope.yaml. This will load the ckpt whose path is exactly the federate.save_to term in llama_modelscope.yaml. As for the training dataset and evaluation dataset, of course, they should not be the same, but may be related. For more details, you can refer to our paper FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning. :)

[ZhouZhanJi]

When you use eval_for_gsm8k/eval.py to evaluate your LLM, it already contains a loading of the evaluation dataset, which is "gsm8k_test.jsonl", thus the usage of the provided yaml file is to provide the model information. In this case, you should run the code python federatedscope/llm/eval/eval_for_gsm8k/eval.py --cfg federatedscope/llm/baseline/llama_modelscope.yaml. This will load the ckpt whose path is exactly the federate.save_to term in llama_modelscope.yaml. As for the training dataset and evaluation dataset, of course, they should not be the same, but may be related. For more details, you can refer to our paper FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning. :)

Ok, it's very nice for you to answer my question, I will try it again by following you advice. Thank you!