Closed smartliuhw closed 1 month ago
How reproducible is this? I am wondering if it has anything to do with https://github.com/huggingface/transformers/pull/29285. Maybe try upgrading your transformers version?
How reproducible is this? I am wondering if it has anything to do with huggingface/transformers#29285. Maybe try upgrading your transformers version?
I tried to run the script again just now and met the same situation again, the loss was around 40 in the beginning
My transformers version is 4.41.2, which should be the latest version(or latest a week ago). Gemma-7B does take more GPU memory than other 7B models, so I think my version of transformers did fixed the issue. Even if it's true that there's some problem with the RoPE, it still can't explain why Gemma-2B works well😂
Fair point. Maybe cc @danielhanchen since he is an expert on Gemma-7B fixes 😂
Fair point. Maybe cc @danielhanchen since he is an expert on Gemma-7B fixes 😂
Hi, I have done some tests and got some findings.
It's important for Gemma model to add a
@vwxyzjn Thanks for tagging me :) Hi :)
@smartliuhw Oh yes we noticed the same issue with packing=True
causing high losses in our blog: https://unsloth.ai/blog/gemma-bugs
The most important caveat for finetuning is you must add the
token (red loss). The blue loss is no token. Packing with TRL works, but has a higher base loss - it’s possible Gemma does not use the T5 packing trick anymore! See the T5 paper page 12 or TensorFlow
@vwxyzjn Thanks for tagging me :) Hi :)
@smartliuhw Oh yes we noticed the same issue with
packing=True
causing high losses in our blog: https://unsloth.ai/blog/gemma-bugs
The most important caveat for finetuning is you must add the token (red loss). The blue loss is no token. Packing with TRL works, but has a higher base loss - it’s possible Gemma does not use the T5 packing trick anymore! See the T5 paper page 12 or TensorFlow
Thanks a lot for your reply! It's true that Gemma does not use the trick you mentioned, I manually print part of the input_ids and decode them, the results are following:
May I ask how can I fix those problems? Or I should use unsloth to train my model(btw, I didn't find the doc for multi-gpu train using unsloth, could you please show me some examples?)
Thanks a lot 🥰
Despite adding <bos>
token and loading model and tokenizer using unsloth, I am still getting very high loss.
Single sequence example -
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/gemma-7b-it-bnb-4bit",
max_seq_length = 2048,
dtype = torch.bfloat16,
load_in_4bit = True,
)
text = """<bos><start_of_turn>user
How many days are there in a week?<end_of_turn>
<start_of_turn>model
There are a total of 7 days in a week. They are Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, and Saturday.<eos>"""
input_ids = tokenizer(text, return_tensors="pt")['input_ids'].to('cuda')
labels = input_ids.clone()
labels[:, :-1] = labels[:, 1:].clone()
labels[:, -1] = -100 # We use -100 to mask the loss at the last position
with torch.no_grad():
outputs = model(input_ids, labels=labels)
loss = outputs.loss
loss.item() # loss is -> 38.96431350708008
cc: @danielhanchen
Hi @danielhanchen, Isn't it better to use group_by_length to group short sequences if packing is disabled?
@kdcyberdude Oh you're not supposed to do labels[:, :-1] = labels[:, 1:].clone()
--> Gemma and Unsloth internally already does that, so you're doing it twice now. Yes group by length is good
@smartliuhw Sorry currently we don't have multi GPU - it'll come in a future release!
Got it @danielhanchen, it does reduce the loss, but it's still 15.05
. way more than what it supposed to be.
@danielhanchen Looking forward to it!
@kdcyberdude I have tried to set packing=False
, got the similar result with you. If you have any progress on that, could you please tell me how to solve it, I would really appreciate that❤️
Hey @smartliuhw, I was able to obtain the following loss curve. It starts at 2.5, increases to 6, and then converges while fluctuating a lot. I'm utilizing the openchat variant of Gemma fine-tuning for low-resource languages, specifically Panjabi, with a batch size of 7*4=28 and a learning rate of 1e-5. You can anticipate better performance when using English datasets.
The gradients are high as well -
Hi @kdcyberdude, thanks a lot for your sharing! I will try to use the variant you mentioned, maybe there's just some problem with Gemma-7b base😂. I have checked their hf homepage and found lots of discussions unsolved
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
I was trainging Gemma-2B and Gemma-7B using sfttrainer, with
Gemma-2B:
![image](https://github.com/huggingface/trl/assets/79445290/364ae2b9-7015-4df6-903b-0a39c8a8c303)
packing=True
set. The Gemma-2B's loss was quite normal, but Gemma-7B's was abnormally high. I'm not sure why this would happen, since both of the model should use the same tokenizer. Gemma-7B:environment dependency: CUDA version: 12.4 torch version: 2.3.0 trl version: 0.8.6
launch command:
tranier set: