Support finding encodings for fine-tuned models

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust

MIT License

125 stars 26 forks source link

Closed spiderbites closed 9 months ago

spiderbites commented 1 year ago

This PR updates encoding_for_model to support fine-tuned models.

I've also updated the model to encoding name mapping based on what's now in the python library.