search

pkoukk / tiktoken-go

go version of tiktoken
MIT License
601 stars 67 forks source link

tiktoken: add custom constructor #40

Closed bobheadxi closed 9 months ago

bobheadxi commented 9 months ago

This change exports a constructor that can be used with custom configuration, e.g. non-OpenAI models.