Use lm/preprocessing.py
to convert a directory with normalized text data into a dataset for
either a character language model or a word language model (see example in its main).
Use lm/trainer.py
to train a character language model or a word language model
(see example in its main). To use real LM and TRAIN hyper-parameters, use CHAR_LM_OPTIONS
,
CHAR_TRAIN_OPTIONS
and WORD_LM_OPTIONS
, WORD_TRAIN_OPTIONS
from lm/configs.py
. These
are based on actual language model training from the literature.