Whisper Tokenizer and `added_tokens`

huggingface / swift-transformers

Swift Package to implement a transformers-like API in Swift

Apache License 2.0

718 stars 80 forks source link

Closed pcuenca closed 11 months ago

pcuenca commented 11 months ago

Tokenizer tests pass, so I'll merge for simplicity. We need to:

Better test WhisperTokenizer in the wild
Handle specialTokens appropriately, they are currently ignored. This affects all tokenizers.