example/main.go run error

[Crisescode]

i finish to make build, but when i test example/main.go, the following error was found:

➜  tokenizers git:(main) ✗ go run example/main.go
---  invalid argument
panic: invalid argument

goroutine 1 [running]:
main.main()
        /home/zhaopanpan/aigc/tokenizers/example/main.go:13 +0x396
exit status 2

and i example/main.go context:

package main

import (
        "fmt"

        "github.com/daulet/tokenizers"
)

func main() {
        tk, err := tokenizers.FromFile("/home/xxx/aigc/tokenizers/test/data/cohere-tokenizer.json")
        if err != nil {
                fmt.Println("--- ", err)
                panic(err)
        }
        // release native resources
        defer tk.Close()
        fmt.Println("Vocab size:", tk.VocabSize())
        // Vocab size: 30522
        fmt.Println(tk.Encode("brown fox jumps over the lazy dog", false))
        // [2829 4419 14523 2058 1996 13971 3899] [brown fox jumps over the lazy dog]
        fmt.Println(tk.Encode("brown fox jumps over the lazy dog", true))
        // [101 2829 4419 14523 2058 1996 13971 3899 102] [[CLS] brown fox jumps over the lazy dog [SEP]]
        fmt.Println(tk.Decode([]uint32{2829, 4419, 14523, 2058, 1996, 13971, 3899}, true))
        // brown fox jumps over the lazy dog
}

[daulet]

@Crisescode did you make build? Perhaps you ended up with older version of libtokenizers.a library. Also, does the json file exist?

[wuchaowei2012]

still encounter the same problem. panic: invalid argument. Even if I rebuild many times