Support loading tokenizer from local folder #76

huggingface / swift-transformers

Swift Package to implement a transformers-like API in Swift

Apache License 2.0

657 stars 70 forks source link

Closed vinu-vanjari closed 6 months ago

vinu-vanjari commented 6 months ago

Resolves #76

Added new method in AutoTokenizer to load from local model folder.
Also added supporting unit tests at Tests/TokenizersTests/FactoryTests.swift
Although we could have simply loaded tokenizer_config.json and tokenizer.json files directly into configurations, I chose to go via LanguageModelConfigurationFromHub because it has it's own magic for choosing appropriate tokenizer class and it also selects fallback tokenizer if needed refer getter var tokenizerConfig: Config?

pcuenca commented 6 months ago

Something's up with the CI, but the tests pass locally so I'll merge. Thanks a lot @vinu-vanjari!