search

Alibaba-NLP / ACE

[ACL-IJCNLP 2021] Automated Concatenation of Embeddings for Structured Prediction
Other
298 stars 44 forks source link

Memory requirements for a new model #55

Open DanrunFR opened 1 year ago

DanrunFR commented 1 year ago

Hello, I am trying to train a new ACE model on a 3.5m word corpus. I want to use the same embedding configuration of the CoNLL 2003 model. I have tried a few times, but I kept running into memory problems. I had about 250G CPU memory but it didn't even get through 3 embedding models. I was wondering how much memory would you recommend using to train my model ?

DanrunFR commented 1 year ago

And also how much time would the training take ? I'm asking because I'm using our lab's computation facility, and I need to reserve the resources I need.

Thank you in advance for your response.

wangxinyu0922 commented 1 year ago

Hi, to run conll 2003 dataset, it takes about 40~60G CPU memory. It seems that a 3.5m word corpus is about 10 times larger than the conll 2003 dataset. So it takes about 400G cpu memory to run the code. I suggest you may remove the flair embeddings (especially the multilingual embeddings) to reduce the feature size.

wangxinyu0922 commented 1 year ago

For training, it takes about 2~4 days to train conll 2003 dataset on a single Tesla V100 GPU. You may reduce the training epoch to reduce the training time.