Return token_strings in Client.tokenize() when offline=True

cohere-ai / cohere-python

Python Library for Accessing the Cohere API

https://docs.cohere.ai

MIT License

278 stars 60 forks source link

Return token_strings in Client.tokenize() when offline=True #494

Open yifanmai opened 3 months ago

yifanmai commented 3 months ago

Fixes #493

CLAassistant commented 3 months ago

All committers have signed the CLA.

abdullahkady commented 3 months ago

Thanks @yifanmai for the contribution!

We intentionally skipped this when we introduced offline local tokenization; since we were optimizing for time. However, let me check how much overhead this adds (for bigger inputs) and if it's negligible we can add it.