search

VinAIResearch / PhoBERT

PhoBERT: Pre-trained language models for Vietnamese (EMNLP-2020 Findings)
MIT License
651 stars 92 forks source link

Error tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base") #26

Closed trungtruc123 closed 3 years ago

trungtruc123 commented 3 years ago

I run pretrain PhoBERT error . ValueError Traceback (most recent call last) in () 3 4 phobert = AutoModel.from_pretrained("vinai/phobert-base") ----> 5 tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base") 6 7 # INPUT TEXT MUST BE ALREADY WORD-SEGMENTED!

/usr/local/lib/python3.6/dist-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, *kwargs) 323 if tokenizer_class is None: 324 raise ValueError( --> 325 "Tokenizer class {} does not exist or is not currently imported.".format(tokenizer_class_candidate) 326 ) 327 return tokenizer_class.from_pretrained(pretrained_model_name_or_path, inputs, **kwargs)

ValueError: Tokenizer class PhobertTokenizerFast does not exist or is not currently imported.****

datquocnguyen commented 3 years ago

For transformers v4.x+:

tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base", use_fast=False)

For transformers v3.x:

tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base")

See readme.md for more details.

trungtruc123 commented 3 years ago

thanks you bro.!

Vào Th 5, 10 thg 12, 2020 vào lúc 12:26 Dat Quoc Nguyen < notifications@github.com> đã viết:

Closed #26 https://github.com/VinAIResearch/PhoBERT/issues/26.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/VinAIResearch/PhoBERT/issues/26#event-4093238353, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKFXDRGNGDNTSHXAYGNYK6TSUBLYPANCNFSM4UUNOU6A .