search

sewenew / tokenizer

C++ implementation of tokenizers, including tiktoken.
Apache License 2.0
11 stars 1 forks source link

can i use this tokenizer for opeanai-whisper ? whsiper using the gpt-2.tiktoken file..... #3

Closed Jaionfire closed 4 months ago

sewenew commented 4 months ago

Of course, you can use it.