search

ufal / udpipe

UDPipe: Trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files
Mozilla Public License 2.0
364 stars 77 forks source link

Too slow training #88

Closed alvelvis closed 5 years ago

alvelvis commented 5 years ago

Is it normal that UDPipe takes more than 4 days to train a model (for Portuguese) using simply "./udpipe --train newmodel.udpipe golden.conllu", in a 8gb RAM i5 CPU machine?

We only managed to finish training a model once, but the result was corrupted: "Loading UDPipe model: Cannot load UDPipe model 'Experimento Revisão de POS/mais-antiga.udpipe'!". Other times we try it, the machine either shuts down or gets stuck in iteration 1 or 2. Already tried in different machines with similar specs, though, all of them under Ubuntu...

Any advice we should follow?

jwijffels commented 5 years ago

I don't have that experience. I trained a UDPipe model with the R interface in january 2018 on Portuguese with the following code and hyperparameters and that trained from 2018-01-11 16:27:25 UTC to 2018-01-12 08:53:49 UTC - so about 16hours. Code here and logs of the process here: https://github.com/bnosac/udpipe.models.ud/tree/master/src/portuguese

foxik commented 5 years ago

Such a slow training is not usual (all UD 2.3 released models take at most a day to train, but we are talking about 20+20 iterations of the tagger and 30 of the parser, so you can observe progress) -- usually, it is caused by unusual XPOS or lemma tags, because all encountered UPOS/XPOS/Lemma triples of a single form are always considered.

I remember that in UD 2.0, the Portugese model used complex XPOS tags, which were causing training to be extremely slow (but in UD 2.2 and 2.3 the XPOS tags of all Portugese treebanks are different). I added dictionary_max_form_analyses tagger option and set it to 10 for the UD 2.0 Portugese treebank, after trying a coarse grid search -- if you are using similarly complex XPOS tags, try using --tagger=dictionary_max_form_analyses=10.

alvelvis commented 5 years ago

Thank you all, especially for the fast answers! Indeed, I removed XPOS column from the train material and the model trained just fine, took me around 16 hours as well. Newer Bosque versions already removed XPOS column, but the experiment we're doing needed us to train on older version...

Thanks again!