search

huggingface / swift-transformers

Swift Package to implement a transformers-like API in Swift
Apache License 2.0
657 stars 70 forks source link

Gemma tokenizer, fix Unicode split #52

Closed pcuenca closed 7 months ago

pcuenca commented 7 months ago

Fixes #51, reported by @davidkoski.

Also had to disable omitting empty sequences to make it work like Python and JavaScript (" " is now split to ["", ""] rather than [])