Is it possible to join the implementation of GPT3 Tokenizer

[OneSeven]

Use Go to implement this function: https://platform.openai.com/tokenizer

[sashabaranov]

Related: https://github.com/openai/tiktoken

[OneSeven]

Related: https://github.com/openai/tiktoken

Thanks, but I think I need a library that can be called through golang.

[sashabaranov]

@OneSeven sure, I mean, we either would need to be able to embed this library (via cgo or otherwise) or would need to translate it from Rust to Go.

[OneSeven]

@OneSeven sure, I mean, we either would need to be able to embed this library (via cgo or otherwise) or would need to translate it from Rust to Go.

Do you have plans to add this functionality to the current SDK. I would love to contribute, but my level is far from enough, sorry.

[sashabaranov]

There's no plan for that right now, but we are open for contributions 😄

I guess you can also call github.com/openai/tiktoken as a separate binary from Go.

[ealvar3z]

@OneSeven sure, I mean, we either would need to be able to embed this library (via cgo or otherwise) or would need to translate it from Rust to Go.

Isn't this library in Python? and if porting; how would you prefer the scaffolding of the porting into your repo? would it be a separate repo and then you import it into go-gpt3, etc. In other words, I am attempting to see your vision if porting it from Python to Go is feasible.

[marcel]

There's a go library already: https://github.com/samber/go-gpt-3-encoder

[OneSeven]

There's a go library already: https://github.com/samber/go-gpt-3-encoder

This library can only be used for English characters, and the correct results cannot be obtained for other languages

[sashabaranov]

@ealvar3z It's Rust wrapped in Python https://github.com/openai/tiktoken/blob/main/src/lib.rs

If it would be possible to bring tokenization with zero (or minimal) dependencies — I'm all for merging it. Otherwise, I think it makes sense to implement it in a separate repo.

[vvatanabe]

Good example of how to count tokens:

• https://github.com/pkoukk/tiktoken-go#counting-tokens-for-chat-api-calls
• https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

[GwynethLlewelyn]

Since the original issue was opened, there has been some progress!

The documentation on the official OpenAI repository currently points to pkoukk/tiktoken-go as the Go library for tokenizing (no endorsements, just a link).

You can see from the test script that it deals with tokens in different languages and alphabets. It might still get things wrong, but at least they are as wrong as the official OpenAI Python version!

Dependencies currently listed by its go.mod:

module github.com/pkoukk/tiktoken-go

go 1.19

require (
    github.com/dlclark/regexp2 v1.10.0
    github.com/google/uuid v1.3.0
    github.com/stretchr/testify v1.8.2
)

require (
    github.com/davecgh/go-spew v1.1.1 // indirect
    github.com/pmezard/go-difflib v1.0.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)

It's not "zero" dependencies as you'd prefer, but close! I haven't looked into the code very deeply.

The dependency upon google/uuid is pretty standard; one wonders why the Go core developers haven't incorporated it into the Go Standard Library yet (it does have a few quirks, though, but because it comes from Google itself, I guess it's ok to use).

The inclusion of dlclark/regexp2 — as opposed to using the standard regexp built on top of Google's RE2 engine — is very likely because the former closely follows the algorithm used by .NET, which might be a requirement for the tokenizer to come up with the same results as tiktoken.

And stretchr/testify is evidently only used for the testing bits; it has no relevance to the overall tokenizer code itself.

Performance, according to the published benchmarks (e.g., those included in its test suite), seems to be the same as the original Python code.

I think you've got your tiktokenizer candidate! 😀