search

chengchingwen / Transformers.jl

Julia Implementation of Transformer models
MIT License
526 stars 75 forks source link

Tokenizer for MPNET model #135

Closed MNLubov closed 1 year ago

MNLubov commented 1 year ago

Hi @chengchingwen,

Thanks for your fantastic package. I use it a lot. I was wondering if you've ever considered adding a tokenizer for the MPNet transformer model? This tokenizer is used in the most powerful sentence transformer model based on MPNET architecture. MPNET tokenizer is a little bit different from BERT tokenizer, but it inherits from the BERT tokenizer.

chengchingwen commented 1 year ago

Should be workable with v0.2.4.