Support finding encodings for fine-tuned models

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust

MIT License

109 stars 26 forks source link

Closed spiderbites closed 4 months ago

spiderbites commented 8 months ago

This PR updates encoding_for_model to support fine-tuned models.

I've also updated the model to encoding name mapping based on what's now in the python library.