LysandreJik
commented
3 years ago Hello @alexvaca0, thank you for opening an issue! Is there a way for you to provide a collab with a reproducer so that we may take a look at the memory issue?
Regarding the very bad performance, and your query that you hope the "architecture itself is not wrongly coded" - rest assured, the architecture was contributed by the author of the model. I believe DeBERTa has historically been hard to pretrain, as I've heard similar reports in the past. Pinging @BigBird01, the author of the model.
Pengcheng, do you have some tips regarding pretaining the DeBERTa model?
I believe the original repository also contains code for model pretraining: https://github.com/microsoft/DeBERTa Have you taken a look at the pretraining script in that repository?
BigBird01
avacaondata
sgugger
github-actions[bot]
Environment info
transformersversion: 4.6.0.dev0Who can help
@patrickvonplaten @LysandreJik @sgugger
Information
Model I am using (Bert, XLNet ...):
I am using a Deberta-base. First I've pre-trained it on >630M texts in spanish, with a BPE tokenizer trained on the same corpus, which in total is 590M (I've performed more than one epoch), using MLM-WWM. Then, I'm trying to use this model on fine-tuning, but I'm facing some issues. First of all, Deberta is supposed to be much better than Bert and Roberta, however I'm experiencing a very bad performance, when compared to the other spanish model: dccuchile/bert-base-spanish-cased (BETO from now on), which supposedly has a worse architecture and which is trained only slightly more than my model. I've tried with many different hyperparameters, following the recommendations in Deberta paper, without improving results. For reference, in a problem where BETO (the model above) achieves 0.97, I'm achieving 0.91 at best. Moreover, as I'm training many models for hyperparameter search (without using your hyperparameter search api), I see that with each new Deberta model the GPU memory usage increases, which doesn't happen with BETO. I think this is a memory leak in the implementation of Deberta, or at least in the token classification and sequence classification layers of deberta. I don't know if this inefficient implementation leading to this memory leak can have any relationship with the poor performance of the model. Could you please take a look at it?
I hope the architecture itself is not wrongly coded, because otherwise we've spent thousands of dollars in training a spanish model from scratch for nothing. Please, if I can give any further information that can help in clear this out, let me know. I'm a little anxious over here because the results aren't as expected and because there are clear signs that the Deberta implementation has, at least, a memory management problem.
The problem arises when using:
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
Expected behavior
it is expected, given the results reported on Deberta paper, that Deberta-base works better than Bert-base with less training (the architecture of BETO), therefore I wouldn't expect that after training for almost as long as BETO we have much worse results than it. Also, it is expected that after each run with trainer, and after deleting the trainer from memory with del Trainer, and releasing gpu memory with torch.cuda.empty_cache(), the gpu memory usage is not increased from run to run, as with other model architectures this doesn't happen, and with Deberta it does.