search

huggingface / swift-transformers

Swift Package to implement a transformers-like API in Swift
Apache License 2.0
536 stars 46 forks source link

add CohereTokenizer support #80

Closed mzbac closed 3 months ago

mzbac commented 3 months ago

The CoHere tokenizer is using BPETokenizer. I did some quick tests locally and it seems to work fine with the BPETokenizer.