Update
After going through the code base I realized that the tokenizer used for this library should give POS tags, whereas sentencepiece does not. It's just a tokenizer.
Is there any workaround??
Since I am trying to use this library to visualize Hindi language data.
How do I use this library with sentencepiece.
Update After going through the code base I realized that the tokenizer used for this library should give POS tags, whereas sentencepiece does not. It's just a tokenizer. Is there any workaround?? Since I am trying to use this library to visualize Hindi language data.