search

sogou / SogouMRCToolkit

This toolkit was designed for the fast and efficient development of modern machine comprehension models, including both published models and original prototypes.
Apache License 2.0
746 stars 162 forks source link

CoQA in Google Colab - OOM (BertWrapper?) #26

Open My3vilMe opened 5 years ago

My3vilMe commented 5 years ago

Hello i'm currently trying to get the CoQA example running in google colab. Unfortunatly i get a OOM at "train_data = bert_data_helper.convert(train_data,data='coqa')". The colab machines only have 12,7 gb of RAM. When I run the toolkit on my local machine i can see that this process takes up to 14gb of RAM. My Question is, is it possible to reduce the memory usage of the bert data helper (bert wrapper)? (and if, could you tell me where exactly?)

Thank you in advance

jgkimi commented 5 years ago

If it is not about a "GPU" memory issue, one suggestion would be not to extract redundant fields of instances, which the model will not use. I'm not sure whether every extracted field is fully used in the next steps, but I think that checking it could be needed.

My3vilMe commented 5 years ago

Thank you for your answer. From my understanding this is a not GPU related memory issue. I will look into extraction process, thanks for the advice.

My3vilMe commented 5 years ago

A little update. I found how to remove fields from the instances but fortunatly that is not important anymore. After another OOM in Colabs I got a notification if I wand to upgrade to a machine with 26GB RAM, yay!. That pretty much solved the problem and I have been able to make a train run. unfortunatly the 12 hours are not enough to also get the evaluation.

I have another thing I would like to ask thou, concerning the answer prediction. In the squad example of BERT you can get a prediction.json at the end, is there a easy way to let the toolkit generate a prediction file too? I also looked into the "get_best_answer" method and couldn't really get it to work (I am rather unexperienced sorry) can you give me some advice there?

Thank you in advance and thanks for making it easy to get into this complex matter! :)