Open DanrunFR opened 1 year ago
Hi @DanrunFR can you check the token-length of the sentences in your dataset? If you have a few very long sentences, you might not be able to use them for training.
Besides that, I would recommend to either finetune the transformers embeddings or use a Bi-Lstm, but not both, as that usually doesn't work too well.
You can simply pass finetune=False
to the transformer embeddings, so the Bi-LSTM can have a stable input and you will save some GPU memory
Hi, Thank you for your response ! This being part of a comparative study of embedding methods, I can't really modify the training set. I'll have to find a way to make it work even with a couple of long sentences. Although I did not know that by default transformer models are finetuned even with BiLSTM. I'll try passing the finetune parameter, we'll see how it goes !
Describe the bug
I keep running into Cuda OOM problems when training NER models using Transformer embeddings and biLSTM-CRF. The trainer can't even get through 1 epoch. It's weird since I used the same script with much bigger datasets and it always worked.
I put the training part of my script in the
To Reproduce
section, and the error log inLogs and Stack traces
.I launch the script withpython flair_bilstm_crf_cv.py --embed_type trans --embed_path "camembert-base" --load_dates --learning_rate 0.025 --min_learning_rate 0.005 --mini_batch_size 4
My corpus train:dev:test = 5.8k : 300 : 300 I tried with 3 models
camembert-base
bert-multilingual-uncased
andxlm-roberta-base
, always the same problem. I also adjusted themax_split_size_mb
to 128mb as suggested in the error message.This has been bugging me for weeks :( Really appreciate any help from people who've encountered this before.
Thanks in advance !
To Reproduce
Expected behavior
It's supposed to just train I guess ? Maybe with more time since I reduced many parameter values but it should work.
Logs and Stack traces
Screenshots
No response
Additional Context
No response
Environment
Versions:
Flair
0.11.3
Pytorch
1.13.1
Transformers
4.26.1
GPU
True