Open saurabhkumar opened 3 months ago
Hello!
I'm glad to hear that it does work for google-t5/t5-base
, so I agree with you that we're probably dealing with the issue from https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139.
I found out that the original fix was this: https://github.com/huggingface/transformers/pull/9487 These changes have also been propagated to mt5: https://github.com/huggingface/transformers/blob/main/src/transformers/models/mt5/modeling_mt5.py#L576
But there's indeed some folks that still report issues. It's an issue that would need to be fixed in transformers
I think.
Alternatively, you can use bf16=True
if your GPU supports that, or train with full precision instead.
Training using google/mt5-base as the base model with fp16 and the triplet loss on all-nli data (following the example with trainer leads to errors as the loss is zero and grad_norm is nan.
{'loss': 0.0, 'grad_norm': nan, 'learning_rate': 0.0, 'epoch': 0.0}
There were similar problems long ago with these models as mentioned here. I want to try this on a GPU without bf16 support. @tomaarsen : It works for google-t5/t5-base so the fix has been applied to that model. Would it be possible to request for help from someone at Hugging Face to apply the same fix to mt5 (or possibly get information from someone in the aya-101 team if they tried it).