search

huggingface / swift-transformers

Swift Package to implement a transformers-like API in Swift
Apache License 2.0
731 stars 81 forks source link

SentencePiece #109

Open beginner-byte opened 4 months ago

beginner-byte commented 4 months ago

Does it support the vocabulary segmentation trained by SentencePiece?

beginner-byte commented 4 months ago

I have converted the "facebook/m2m100_418M" model, and now it is divided into encoder and decoder. I'm not sure what to do next. Since "facebook/m2m100_418M" does not have a tokenizer.json and uses a SentencePiece tokenizer, I'm a beginner.