MT0021: Training a bilingual tokenizer

[TenzinGayche]

Description :

Training a Bilingual tokenizer with 32k vocab size and low fertility score using sentencepiece library

To dos:

• [x] Train a tokenizer with the existing MT datas
• [x] Create a Tibetan Monolingual data
• [x] Create a English monolingual data
• [x] Train a tokenizer on Mt data + monolingual
• [x] Compare the tokenizer with other existing tokenizer ,using compression rate and fertility score as evaluation metric

[TenzinGayche]

Bo_en Tokenizer V1