Open SandyPanda-MLDL opened 5 months ago
I am using PL-bert and wanted to replace the Transformer-XL tokenizer with the Indicbert tokenizer. But, I was curious to know whether it can be used that way. Whether, Indicbert does phone level tokenization.
I am using PL-bert and wanted to replace the Transformer-XL tokenizer with the Indicbert tokenizer. But, I was curious to know whether it can be used that way. Whether, Indicbert does phone level tokenization.