MikaSie
commented
3 months ago @muellerzr
I have found an issue with accelerate test, it seems like the issue with my training script has something to do with transformers:
I saw that fsdp_transformer_layer_cls_to_wrap is depreciated. It seems like FSDP_TRANSFORMER_CLS_TO_WRAP isn't set in this file. See code snippet:
def set_auto_wrap_policy(self, model):
from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy, transformer_auto_wrap_policy
default_transformer_cls_names_to_wrap = (
",".join(model._no_split_modules) if getattr(model, "_no_split_modules", None) is not None else ""
)
if self.auto_wrap_policy is None:
auto_wrap_policy = os.environ.get("FSDP_AUTO_WRAP_POLICY", "NO_WRAP")
if auto_wrap_policy == FSDP_AUTO_WRAP_POLICY[0]:
transformer_cls_names_to_wrap = os.environ.get(
"FSDP_TRANSFORMER_CLS_TO_WRAP", default_transformer_cls_names_to_wrap
).split(",")
transformer_cls_to_wrap = set()
for layer_class in transformer_cls_names_to_wrap:
transformer_cls = get_module_class_from_name(model, layer_class)
if transformer_cls is None:
raise Exception("Could not find the transformer layer class to wrap in the model.")
else:
transformer_cls_to_wrap.add(transformer_cls)Maybe it is good to remove FSDP_TRANSFORMER_CLS_TO_WRAP ? It isn't imported from .constants while FSDP_AUTO_WRAP_POLICY is imported from constants.
I ran through the code and it seems like that set_auto_wrap_policy works fine when a model is given to it. I've tried by giving it LLama3 as the model and LlamaDecoderLayer is properly returned. So it seems like the issue in my personal training script is not with accelerate but with transformers.
But then, the isssue for accelerate test will still exist. It seems like it fails because the Regression model that is given to accelerator.prepare() doesn't have any _no_split_modules as can be seen here.
Might be good to fix this as running accelerate test won't work properly this way. This gives the false impression that the accelerator config isn't setup properly while in fact it is (probably) set up just fine!
I think this just happens when FSDP is used as this will search for the _no_split_modules. Could be an idea to just pass the regression model a basic BertLayer as the _no_split_modules to make the test pass.
BenjaminBossan
github-actions[bot]
System Info
Information
Tasks
no_trainerscript in theexamplesfolder of thetransformersrepo (such asrun_no_trainer_glue.py)Reproduction
Packages
Code
Expected behavior
tldr;
I'm encountering an error when running
accelerate test. This happens when runningaccelerate testwith and without settingfsdp_transformer_layer_cls_to_wraptoLlamaDecoder, LlamaMLP. The same issue seems to happen when I run my training script:fsdp_transformer_layer_cls_to_wraptoLlamaDecoder, LlamaMLP, I get the same error as runningaccelerate test:Exception: Could not find the transformer layer class to wrap in the model.fsdp_transformer_layer_cls_to_wrapfrom the config file, I get a different error when running my training script:ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more detailsIntroduction
Hi! I'm trying to fine-tune LLama3-8B on a summarization dataset of about 1500 instances. The dataset contains long documents, often over 8K tokens. I want to use FSDP + QLORA to try and finetune LLama3 8B.
I'm following these guides as inspiration: bitsandbytes Guide Phil Schmid Guide Huggingface Accelerate Guide Huggingface PEFT Guide Huggingface Bitsandbytes Guide
First issue:
At first, I didn't set
fsdp_transformer_layer_cls_to_wrapas it isn't defined in the mentioned guides. And with my training script I am able to start training. But, then I'm unable to save the model or push it to the hub after training. I have had a full training run before and could see the loss reducing. The yaml file looks as follows in this scenario:When running :
I get the following output
There is a warning, a print statement and finally the ValueError. The warning is repeated many times and seems to come from here.
It seems like the model isn't wrapped properly or that something is wrong with the quantization, as the ValueError indicates. But I don't think it has anything to do with quantization because when I print my model I see the following:
Second issue:
I couldn't really figure out what this issue is and after a while I decied to check out if my accelerate config was set up properly, so I decided to run
And I get the following error:
I thought the issue was that I didn't set
fsdp_transformer_layer_cls_to_wrap. So after checking out https://github.com/huggingface/accelerate/pull/1947 and https://github.com/tatsu-lab/stanford_alpaca/issues/58 . I figured that this was the issue and setfsdp_transformer_layer_cls_to_wraptoLlamaDecoderLayer, LlamaMLP(Like in the system info above). I thought this would solve the issue but when I run:This error occurs:
This is again the same issue that occurs when running
accelerate testwith and without settingfsdp_transformer_layer_cls_to_wrap:As you can see the issue has something to do with that the transformer layers cannot be found.
I think that this has something to do with how the model is wrapped in FSDP but I'm not sure what is going on.
Please let me know if you have any idea as I'm getting stuck in as of now! I also find it quite odd that
accelerate testdoesn't work with and without the settingfsdp_transformer_layer_cls_to_wrap.