search

pkoukk / tiktoken-go

go version of tiktoken
MIT License
601 stars 67 forks source link

gpt2 support appears missing #41

Open tmc opened 6 months ago

tmc commented 6 months ago

I'm not seeing where the gpt2 model is handled similar to how it is done here: https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py#L10

A simple test like:

+func TestGpt2Encoding(t *testing.T) {
+       if _, err := EncodingForModel("gpt2"); err != nil {
+               t.Error(err)
+       }
+}

will fail like so:

--- FAIL: TestGpt2Encoding (0.00s)
    tiktoken_test.go:36: Unknown encoding: gpt2
FAIL
exit status 1
FAIL    github.com/pkoukk/tiktoken-go   0.194s