tlikhomanenko
commented
4 years ago Hi @samin9796,
1. I want to build a character level LM. Previously I built a word level LM using KENLM and for that I created a text file with one sentence per line and there are thousands of such lines in that file. How to prepare dataset for a char level LM? Is there anything else that is different from building a word level LM?
You need to split additionally each word into characters sequence separated by space. For example, if you had "hello world" sentence to build word-level LM, now to build letter-based LM you need to have "h e l l o | w o r l d |": so space in KenLM is treated as a separator between characters, also we are adding special token | to have words separation. You can do this in a bit different way like "_h e l l o _w o r l d" where you distinguish "start word" letters and letters inside word. This is up to you how to define characters vocabulary.
2\. To prepare data for Convlm, this command is supposed to be used.source prepare_fairseq_data.sh [DATA_DST] [MODEL_DST] [FAIRSEQ PATH]
Here, can you please elaborate what it means by DATA_DST? Is it the path of the input text file that I prepared for the word level LM?
Here is what you need to call https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/lexicon_free/librispeech/prepare_fairseq_data.sh#L9 - to prepare the word-based data (the next command is for char, there is only data differs, parameters you can use the same, even call with or without --padding-factor 1). $FAIRSEQ is path to the fairseq cloned repo, $DATA_DST is prefix to the text corpus and $MODEL_DST path where the processed data will be stored. In your case for another language just call
python "path/to/fairseq/preprocess.py" --only-source \
--trainpref "path/to/your/text/corpus" \
--validpref "path/to/your/valid/text/corpus" \
--testpref "path/to/your/test/text/corpus" \
--destdir "path/to/the/folder/to/store/binarized/fairseq/data" \
--thresholdsrc 10 \
--padding-factor 1 \
--workers 16here thresholdsrc means that words occuring in the text <=10 times will be mapped to unk.
1. I am working with another language, not English. Does convert_convlm.sh work to convert in my case as well?
For converting your model you use in the same way call of save_pytorch_model.py from the https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/lexicon_free/librispeech/convert_convlm.sh and then you need to fix the vocabulary dict (in case you use the same fairseq architecture): 221452 = is vocab size + 4, the same for character LMs 40 = vocab size + 4 (4 is for special fairseq tokens). So if you have 100k vocab during training then you need to put 100004. If you modify arch then you need to fix this one too https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/lexicon_free/librispeech/lm_librispeech_convlm_word_14B.arch. Here is the info about parameters on converting the model https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/utilities/convlm_serializer/Serialize.cpp#L22). Just let me know how do you plan to change fairseq training command then I can navigate you what command to use.
samin9796
I have some questions regarding language model.
I want to build a character level LM. Previously I built a word level LM using KENLM and for that I created a text file with one sentence per line and there are thousands of such lines in that file. How to prepare dataset for a char level LM? Is there anything else that is different from building a word level LM?
To prepare data for Convlm, this command is supposed to be used.
Here, can you please elaborate what it means by DATA_DST? Is it the path of the input text file that I prepared for the word level LM?
Thank you.