search

flairNLP / flair

A very simple framework for state-of-the-art Natural Language Processing (NLP)
https://flairnlp.github.io/flair/
Other
13.89k stars 2.1k forks source link

Out of memory error when training NER model #95

Closed iamyihwa closed 6 years ago

iamyihwa commented 6 years ago

Hello,

I was training NER model.

However I have ran into a memory error when training NER . I have used the default settings from train.py

image

I am using p2.xlarge with 61 GB of RAM.

Since it is gpu running out of memory and not due to flair, I did a quick check of this error and it shows me that data could be too big. However the training/ validation / test set are maximum 3 MB of data.

I am just wondering what type of machine you use to train, and whether you had similar issues.

I just checked the size of embeddings, and character level embeddings. image and they are not too big .. Language model is in the order of 10MB .

iamyihwa commented 6 years ago

I have reduced the minibatch size and the error goes away. I have reduced mini_batch_size = 36 -> 12

iamyihwa commented 6 years ago

Now that I read about the effect of minibatch size, I wonder if this is not too big reducing. Is there any ideal mini batch size that you have explored? https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent