Closed mxa4646 closed 3 years ago
OK, I can reproduce the problem with just google/mt5-small and 2 gpus:
export BS=1; PYTHONPATH=../../src USE_TF=0 deepspeed --num_gpus=2 ./finetune_trainer.py --model_name_or_path google/mt5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size 1 --per_device_train_batch_size 1 --predict_with_generate --eval_steps 25000 --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16
We will get it sorted out today.
ok, the problem had nothing to do with DeepSpeed, it's just a seq2seq neglect.
The fix is:
diff --git a/examples/seq2seq/utils.py b/examples/seq2seq/utils.py
index 8b24bfda..303b89f7 100644
--- a/examples/seq2seq/utils.py
+++ b/examples/seq2seq/utils.py
@@ -563,7 +563,7 @@ def freeze_embeds(model):
"""Freeze token embeddings and positional embeddings for bart, just token embeddings for t5."""
model_type = model.config.model_type
- if model_type == "t5":
+ if model_type in ["t5", "mt5"]:
freeze_params(model.shared)
for d in [model.encoder, model.decoder]:
freeze_params(d.embed_tokens)
Please let me know if you can manage to apply this fix. I will make a proper PR later, but it'll take some work, since I need to make a tiny mt5 model and add a test.
You can just edit the file if you don't know how to apply a patch.
The fix should be merged shortly https://github.com/huggingface/transformers/pull/9879
I can solve the --freeze_embeds
bug now, thanks for your help! @stas00
As for questions 3 and 4, I noticed that the title of the issue has been edited. I don't know if these questions are caused by the model or the seq2seq trainer. Maybe I should raise them in a new issue?
Oh, you wrote those items as steps to reproduce the problem, so I didn't know that those were issues that needed to/could be fixed.
Once I discovered that the issue you posted was unrelated to DeepSpeed I took the liberty to adjust the subject.
In general, yes, let's try to keep each issue separate, so that it makes it much easier to track things and not let things fall between the cracks.
Back to your follow up question:
Looking just at the params:
So the 2nd model is substantially larger, and if t5-3b fit tightly onto a 24GB card it's not surprising that the larger model didn't.
and in addition to model params you also need to allocate memory for:
I tried mt5-xl on 4x 40gb gpu setup and it worked, but took ~29GB on each GPU, so there is the problem - you're 5GB short.
The command I run was:
export BS=1; PYTHONPATH=../../src USE_TF=0 /usr/bin/time -v deepspeed --num_gpus=4 ./finetune_trainer.py --model_name_or_path google/mt5-xl --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size 1 --per_device_train_batch_size 1 --predict_with_generate --eval_steps 25000 --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16
You may try to tweak the buffer sizes in ds_config.json
but I think the gap is too big.
I'm working on a 2D Parallelism solution that will combine pipe|model-parallelism w/ ZeRO-DP (DeepSpeed), which should enable such feats with huge models, but it might take some time. The docs aren't quite there so it takes a lot of trial and error to move forward. You may want to track this PR https://github.com/huggingface/transformers/pull/9765 for updates.
Alternatively when fairscale or DeepSpeed releases ZeRO phase 3, you shouldn't have a problem loading this model onto 4x 24GB gpus. Currently the problem is that the model params are too big w/o phase 3. In phase 3 params are partitioned too - problem solved.
I tried mt5-xl on 4x 40gb gpu setup and it worked, but took ~29GB on each GPU, so there is the problem - you're 5GB short.
That's help a lot! Thank you!
I am also looking forward to ZeRO stage 3 and your pipe|model-parallelism. Hope one day we can working on it. Thank you again!
And I got the overflow problem. This is not surprising me because MT5-large seems not fixed FP16 yet.
Did you get nan
loss or gradient overflow warning ? And yes, fp16 is still not working for mT5-large
I assume that T5-3b and MT5-xl should be in the same order of magnitude
mT5-xl is actually quite bigger than T5-3b for two reasons
gated-gelu
activation instead of relu
. gated-gelu
adds one extra linear layer in every feed-forward layer.@patil-suraj That's very helpful! Thank you a lot!
Now I understand that there are many differences between mT5-xl and T5-3b, and I will set up separate experiments for them in the future. By the way, do you have any plans to repair the FP16 in mt5-large/xl ?
Dear @patil-suraj, here you have mentioned for mt5-small you have made it work with fp16? since you did not mention this model, do you mind telling me how you made it work? I am having a hard time with mt5-small with fp16 thanks a lot for your advice
I have a similar error here
from transformers import T5TokenizerFast, MT5ForConditionalGeneration
tokenizer = T5TokenizerFast.from_pretrained('google/mt5-base') # "google/mt5-base" "google/mt5-large" "google/mt5-xl"
model = MT5ForConditionalGeneration.from_pretrained('google/mt5-base', return_dict=True)
condition = "translate English to German: "
input = "My name is Azeem and I live in India"
# You can also use "translate English to French" and "translate English to Romanian"
input_ids = tokenizer(condition+input, return_tensors="pt").input_ids # Batch size 1
outputs = model.generate(input_ids)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded)
Stacktrace:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
[<ipython-input-8-f9822d331a70>](https://localhost:8080/#) in <module>()
3 tokenizer = T5TokenizerFast.from_pretrained('google/mt5-base') # "google/mt5-base" "google/mt5-large" "google/mt5-xl"
4
----> 5 model = AutoModelForSeq2SeqLM.from_pretrained('google/mt5-base', return_dict=True)
6
7 condition = "translate English to German: "
8 frames
[/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py](https://localhost:8080/#) in __getattribute__(self, key)
250 if key != "attribute_map" and key in super().__getattribute__("attribute_map"):
251 key = super().__getattribute__("attribute_map")[key]
--> 252 return super().__getattribute__(key)
253
254 def __init__(self, **kwargs):
AttributeError: 'MT5Config' object has no attribute 'relative_attention_max_distance'
@stas00 any idea? I'm using HF master:
!pip install git+https://github.com/huggingface/transformers.git
@loretoparisi
This is because T5Config now has relative_attention_max_distance
attribute introduced in the #16155 which was missing from MT5Config
. Fix is here #16170
Environment info
transformers
version: 4.2.2Who can help
@stas00,@patrickvonplaten, @patil-suraj
Information
Model I am using (MT5-xl,MT5-large):
The problem arises when using:
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
exmaples/seq2seq/finetune_trainer.py
, which was originally used to reproduce the training of T5-3b on single 3090. All processes are the same as #8771 and it can reproduce the training of T5-3b(whether single card or 2/4 cards).--freeze_embeds
seems to bring bugs. I used 4*3090, My script isHere is my report:
--freeze_embeds
and tried to train MT5-xl again, but I got CUDA out of memory. My device is 4*24G 3090, with BS=1, ZeRO stage=2, and CPU_offload=true. I assume that T5-3b and MT5-xl should be in the same order of magnitude, and I can do it on t5-3b, so I think this should not happen.Expected behavior