search

pkoukk / tiktoken-go

go version of tiktoken
MIT License
601 stars 67 forks source link

feat: add support for latest embedding models #43

Closed matthiasthomas closed 1 month ago

matthiasthomas commented 4 months ago

Adding support for latest embedding models:

matthiasthomas commented 4 months ago

@pkoukk - Would be happy to merge this one as I just moved to large3 🙏

matthiasthomas commented 2 months ago

@zjy282 @felizalde - Feel free to user my branch if you need, since @pkoukk isn't responsive.

I won't be maintaining though, since I moved to https://pkg.go.dev/github.com/hupe1980/go-tiktoken which is active and supports 3 large/small.