search

edponce / FACET

Framework for Annotation and Concept Extraction in Text
Other
2 stars 0 forks source link

Add SentencePiece, BertTokenizer, and similar #22

Open edponce opened 4 years ago

edponce commented 4 years ago

More tokenizers can be included, see https://huggingface.co/transformers/main_classes/tokenizer.html pip install transformers

from transformers import BertTokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') tokenizer.tokenize("Hello, world!")