How to use scattertext with sentencepiece

JasonKessler / scattertext

Beautiful visualizations of how language differs among document types.

Apache License 2.0

2.23k stars 289 forks source link

How to use scattertext with sentencepiece #48

Closed aman5319 closed 4 years ago

aman5319 commented 4 years ago

How do I use this library with sentencepiece.

Update After going through the code base I realized that the tokenizer used for this library should give POS tags, whereas sentencepiece does not. It's just a tokenizer. Is there any workaround?? Since I am trying to use this library to visualize Hindi language data.

JasonKessler commented 4 years ago

I've added a SentencePiece capability to the new version of Scattertext. See https://github.com/JasonKessler/scattertext#visualizing-sentencepiece-tokens for an example of how to use it.