tokenizer that supports Thai and English
https://drive.google.com/drive/folders/1rUs765_FzalZWOJRSRL0cbQGW3lrV4JM?usp=sharing
tokenizer that supports only Thai
https://drive.google.com/drive/folders/1hJ4jsXdypP4mqZDsEgxEEfO-CLwAzHTj?usp=sharing
lmcut/weight/
python setup.py bdist_wheel
pip install dist/lmcut*
from lmcut import tokenize
text = "โรงแรมดี สวยงามน่าอยู่มากๆ"
result = tokenize(text)
print(result)
Result is a list of tokens:
['โรง', 'แรม', 'ดี', 'สวยงาม', 'น่า', 'อยู่', 'มาก', 'ๆ']
To train a language model, you should provide two .txt files of raw text. One for training and one for development. The model supports only Thai and English texts. Characters from other languages will be treated as unknown characters.
See an example in data/lm_data/
You can define your custom datasets in the function get_path_data_LM
in train/get_corpus.py
.
The dataset name defined there will be use in the command line flag --dataset
.
note: if you use InterBEST2009, the boundary markers must be removed first.
To train a new language model, run
python train/LanguageModel.py \
--dataset [dataset name] \
--batchSize 60 \
--char_dropout_prob 0.01 \
--char_embedding_size 200 \
--hidden_dim 500 \
--layer_num 3 \
--learning_rate 0.0001 \
--sequence_length 100 \
--epoch 10 \
--len_lines_per_chunk 1000 \
--optim [adam or sgd] \
--lstm_num_direction [2 means bidirectional and 1 means uni directional] \
--add_note "..add some note.."
Command example
python train/LanguageModel.py \
--dataset default \
--batchSize 32 \
--char_dropout_prob 0.01 \
--char_embedding_size 100 \
--hidden_dim 100 \
--layer_num 2 \
--learning_rate 0.0001 \
--sequence_length 100 \
--epoch 3 \
--len_lines_per_chunk 100 \
--optim adam \
--lstm_num_direction 2 \
--add_note "test if code runs properly"
To resume the training of a language model, run
python train/LanguageModel.py \
--load_from [model name to resume] \
--dataset [dataset name] \
--epoch 3 \
--learning_rate 0.0001 \
--over_write 1
[language model name]
must starts with LM
, for example, LM_2022-05-03_18.59.59
.
Model artifacts are in train/checkpoints_LM
LM_2022-05-03_18.28.05 (log file)
LM_2022-05-03_18.28.05.json (data about model structure used when reloading the model)
LM_2022-05-03_18.28.05.pth.tar (model weights)
LM_log.csv (a log file in csv, for collecting experiment results)
https://drive.google.com/drive/folders/1QKOctAPYIpC7b3beLGvOJ-h43-T85Yjy?usp=sharing
command (note: ty dataset is not publicly available)
python train/LanguageModel.py \
--dataset ty \
--batchSize 64 \
--char_dropout_prob 0.01 \
--char_embedding_size 200 \
--clip_grad 0.5 \
--hidden_dim 514 \
--layer_num 2 \
--learning_rate 0.0001 \
--sequence_length 150 \
--epoch 20 \
--len_lines_per_chunk 1000 \
--optim adam \
--lstm_num_direction 2 \
--lr_decay 0.01 \
--sgd_momentum 0.02
|
.create_dataset.py
.data/toy
as an example.get_path_data_tokenizer
in train/get_corpus.py
To train a new tokenizer, you could run:
python train/Tokenizer.py \
--dataset default \
--epoch 3 \
--lstm_num_direction 2 \
--batchSize 30 \
--sequence_length 80 \
--char_embedding_size 100 \
--hidden_dim 60 \
--layer_num 2 \
--optim adam \
--learning_rate 0.0001
To load a pre-trained language model(the embedding layer and recurrent layer) to the tokenizer and train, you could run
python train/Tokenizer.py \
--load_from [language model name] \
--dataset default \
--epoch 2 \
--learning_rate 0.0001 \
--over_write 0
[language model name]
should begin with LM
, for example, LM_2022-05-03_18.59.59
.To resume the training of a tokenizer, you could run
python train/Tokenizer.py \
--load_from [tokenizer name] \
--dataset default \
--epoch 2 \
--learning_rate 0.0001 \
--over_write 1
[tokenizer name]
should begin with Tokenizer
, for example, Tokenizer_2022-05-03_20.46.35
.
Use --over_write 1
if you want to replace the loaded model with the trained model.
With --over_write 0
it will save the trained model as a separate model.
Model artifacts are in train/checkpoints_tokenizer
Tokenizer_2022-05-03_18.53.26 (log file)
Tokenizer_2022-05-03_18.53.26.json (data about model structure used when reloading the model)
Tokenizer_2022-05-03_18.53.26.pth.tar (model weights)
tokenizer_result.csv (a log file in csv, for collecting experiment results)
For more detail about other arguments, see train/Tokenizer.py
and train/LanguageModel.py
data/news_00001.txt
and data/TEST_100K.txt
are from InterBEST2009 corpus by NECTEC
The project is funded by TrustYou. The author would like to sincerely thank TrustYou and other contributors.
All original code in this project is licensed under the MIT License. See the included LICENSE file.