Closed Polaris-JZ closed 7 months ago
The yaml you used is only a testcase, could you please try configs in https://github.com/alibaba/FederatedScope/tree/llm/federatedscope/llm/baseline/exp_yaml/alpaca with tuned hyperparameters?
Thanks for your advice.
I have tried https://github.com/alibaba/FederatedScope/blob/llm/federatedscope/llm/baseline/exp_yaml/alpaca/alpaca_federate.yaml, the test loss become lower, but train loss also fluctuates a lot.
Additionally, I try to create another dataset followed the format of alpaca, the same thing happens: the loss is not decreasing/convergent, and the test loss is very high (like 3000). Could you please help me to give some advice?
Assuming your dataset is good enough, you can try to adjust the following hyper-parameters:
Thanks for your advice. I have another problem. When I'm trying to use the deepspeed accelation. The error rises: KeyError: 'Non-existent config key: llm.accelation'. My config is:
Sorry for the outdated document. Please use the following configs to setup Deepspeed (for other usage, please refer to https://github.com/alibaba/FederatedScope/blob/llm/federatedscope/core/configs/cfg_llm.py):
# ---------------------------------------------------------------------- #
# Deepspeed related options
# ---------------------------------------------------------------------- #
cfg.llm.deepspeed = CN()
cfg.llm.deepspeed.use = False
cfg.llm.deepspeed.ds_config = ''
We'll fix it ASAP.
Thanks for your reply. I was wondering if there was any method to enable multi-gpu training for a client or under centralized training setting.
You can set cfg.train.data_para_dids = [] # torch.nn.DataParallel devices
to enable DataParallel training.
Hi,
I use your config llama.yaml to conduct federated training, but the training log shows that the loss is not decreasing/convergent, and the test loss is very high. I was wondering if there are any problems.
I only change two places in the llama.yaml: