search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
135.02k stars 27.02k forks source link

Unable to Finetune Deberta #19894

Closed ra-MANUJ-an closed 1 year ago

ra-MANUJ-an commented 2 years ago

I am trying to finetune deberta for irony detection task, colab's notebook link can be found here

When I try to use 'microsoft/deberta-v3-base' checkpoint with AutoModel, I'm getting the following error :

RuntimeError: Expected target size [32, 2], got [32] but when I use the same model with 'bert-base-uncased' or roberta (with some changes in head) it works fine. The one can find working code for bert based in this notebook.

When I printed the shapes of predictions and labels, I got outputs as torch.Size([32, 30, 2]), torch.Size([32]) respectively. In the case of bert, shapes of outputs were torch.Size([32, 2]), torch.Size([32]) for predictions and labels.

Here 32 is the batch size, and 30 is the sequence length.

Can someone let me know what I'm doing wrong?

ra-MANUJ-an commented 2 years ago

@sgugger @patil-suraj @patrickvonplaten

sgugger commented 2 years ago

Please use the forums to get help debug your code. In this instance you are using the base pretrained model (without a classification head) to do classification so it does not work. You should consider using AutoModelForSequenceClassification`.

ra-MANUJ-an commented 2 years ago

okay, sure will take care from next time and thanks for the response! Just one question, do bert and roberta provide classification heads in their base models?

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.