Closed Longyichen closed 9 months ago
It seems to be a lack of support for native FSDP for these settings. Since our repo does not use transformers code, we probably cannot easily use the PR? Plus, Composer has another wrapper on top of FSDP which might add extra complexity.
I am currently encountering this problem when running a pruning experiment on a single GPU. I want to solve it by turning off FSDP, but this code seems to be incompatible with settings without FSDP and encounters many problems. But when I turned on the FSDP settings, this problem prevented me from getting Checkpoint. Are you able to share the general configuration of your machine, such as cuda, FSDP and other infrastructure? Will you encounter this problem when running a program on a single GPU?
Hiii I never experimented with single GPUs. You might need to deactivate the FSDP setup in the yaml scripts to completely turn off FSDP by setting`fsdp_config=null.
cuda 12.2
but I think cuda 12.0
and cuda 11.7
should work as well.
This problem occurs when the pruning program is finished and saved. This issue results in no way to save checkpoints normally
There are related discussions about this problem on the Internet https://github.com/huggingface/transformers/issues/24874 https://github.com/huggingface/transformers/issues/24874 , and it seems to have been fixed in the transformer library. Are you considering updating the version or adopting other fixes?