Fix bug that always downloads tokenizer files.

Unbabel / OpenKiwi

Open-Source Machine Translation Quality Estimation in PyTorch

https://unbabel.github.io/OpenKiwi/

GNU Affero General Public License v3.0

229 stars 48 forks source link

Fix bug that always downloads tokenizer files. #105

Open yym6472 opened 3 years ago

yym6472 commented 3 years ago

When creating XLMRobertaTextEncoder object, the tokenizer name will be rewritten to xlm-roberta-base if a local model path is configured, so that the framework will always download the tokenizer files via the Internet (#102). This PR is to fix that.

CLAassistant commented 3 years ago

All committers have signed the CLA.