Any benchmark for throughput?

pkoukk / tiktoken-go

go version of tiktoken

MIT License

601 stars 67 forks source link

Any benchmark for throughput? #14

Closed WesleyYue closed 1 year ago

WesleyYue commented 1 year ago

Curious how this compares.

philippgille commented 1 year ago

Against non-Go tokenizers or other Go ones?

WesleyYue commented 1 year ago

Both

pkoukk commented 1 year ago

Sorry, I am currently unable to provide benchmark testing results. The performance of the program is highly correlated with the language being tested. I have not yet find sufficiently representative test data that could fully evaluate the performance across different languages.

pkoukk commented 1 year ago

I tested encoding Universal Declaration of Human Rights with different languages, here is the result. Benchmark