Can I replace it with a trained SentencePiece model?

studio-ousia / luke

LUKE -- Language Understanding with Knowledge-based Embeddings

Apache License 2.0

705 stars 101 forks source link

Open mumumu09chi opened 1 year ago

mumumu09chi commented 1 year ago

example：spm_tokenizer = MLukeTokenizer.sp_model("./model/sentencepiece.bpe.model") Is there a way to change it?

ryokan0123 commented 1 year ago

We haven't tried that. It seems to be in the scope of the transformers library so I recommend referring to other related documents.