search

ThilinaRajapakse / simpletransformers

Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conversational AI
https://simpletransformers.ai/
Apache License 2.0
4.08k stars 727 forks source link

xlm-roberta-bert is taking lot of time and multiprocessing is disabled as well #398

Closed suryapa1 closed 4 years ago

suryapa1 commented 4 years ago

Describe the bug

Even for 100 documents training is taking more than expected time like 20+hrs for xlm-roberta-large as model_type, Is there any workaround for this to work

suryapa1 commented 4 years ago

Why this functionality is disabled: UserWarning: use_multiprocessing automatically disabled as xlmroberta fails when using multiprocessing for feature conversion. f"use_multiprocessing automatically disabled as {model_type}"

ThilinaRajapakse commented 4 years ago

The warning states why it's disabled. Because using multiprocessing generates an error with xlmroberta for some reason.

The training time depends on a lot of factors like the size of the model, the type of the model, the GPU you are using, etc. So it's impossible for me to say why it's taking 20+ hours or even whether that's unexpected.

suryapa1 commented 4 years ago

Model am using is xlm-Roberta-base with 17000 documents for now with 17 labels , gpu is p2x,16gb gpu memory , any suggestion for me on mode type of xlm-bert as model has to support english and German language

suryapa1 commented 4 years ago

Does xlm-Roberta supports TPU, If so how to enable tpu on the same

ThilinaRajapakse commented 4 years ago

Do you have very long documents? 17000 sample dataset shouldn't take that long to train.

Simple Transformers doesn't have native TPU support yet.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.