search

benob / recasepunc

Model for recasing and repunctuating ASR transcripts
BSD 3-Clause "New" or "Revised" License
126 stars 20 forks source link

what way is used for Tokenize the dataset? #19

Open cdgraff opened 11 months ago

cdgraff commented 11 months ago

Doc, say: All models are trained from the 1st 100M tokens

Can share some example how prepare that 100M tokens from the text input?, I'm trying to train support for ES lang.

Thanks

benob commented 11 months ago

The tokenizer is included in the recasepunc.py script. It is a modified version of the BERT tokenizer which keeps punctuation when lower-casing. Example usage:

xzcat cc-100/$lang.txt.xz | python recasepunc.py --lang=$lang preprocess 100000000 > data/$lang-100M.txt