kriskrisliu
commented
11 months ago The script is correct. Please show us the complete error logs.
Closed WeiXuanLi-1024 closed 10 months ago
kriskrisliu
commented
11 months ago The script is correct. Please show us the complete error logs.
WeiXuanLi-1024
commented
11 months ago kriskrisliu
commented
11 months ago Did you modify any codes?
WeiXuanLi-1024
commented
11 months ago 没有改动任何代码
Did you modify any codes?
WeiXuanLi-1024
commented
11 months ago 使用的单GPU
WeiXuanLi-1024
commented
11 months ago 这个问题已经解决：参考https://github.com/huggingface/accelerate/pull/1745/commits/940ae8dfff504ce7e7e3015bcbbd17e3e8cd3157
原因： # FSDP raises error when single GPU is used with offload_to_cpu=True for FULL_STATE_DICT
也就是在使用单GPU时，必须要将本仓库中 misc.py line348 两个参数设置为 rank0_only=False, offload_to_cpu=False设置完运行结果：正常保存
funkyyyyyy
commented
11 months ago 这个问题已经解决：参考huggingface/accelerate@940ae8d
原因： # FSDP raises error when single GPU is used with
offload_to_cpu=Truefor FULL_STATE_DICT # so, only enable it when num_processes>1 也就是在使用单GPU时，必须要将本仓库中 misc.py line348 两个参数设置为 rank0_only=False, offload_to_cpu=False设置完运行结果：正常保存
just clarifying, if using single GPU, set https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/afa3947742d6262255ba8ba1808ff999246e4abd/accessory/util/misc.py#L347C29-L347C29 offload_to_cpu=False
thank you @WeiXuanLi-1024
我使用 alpacaLlava_llamaQformerv2Peft_QF_13B.sh 微调模型，在迭代完一次epoch ， 保存模型时出现了错误，使用单卡跑的。脚本内容如下
File "/home/liwx/anaconda3/envs/accessory/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/liwx/anaconda3/envs/accessory/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py", line 171, in _unshard_fsdp_state_params _validate_unshard_params_args( File "/home/liwx/anaconda3/envs/accessory/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py", line 140, in _validate_unshard_params_args raise NotImplementedError( NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet