Closed transitive-bullshit closed 6 months ago
Disclaimer: I'm working on https://github.com/dqbd/tiktoken
Hi! I have extended the compare-tokenizers
benchmark to include tiktoken-node
and directly compare @dqbd/tiktoken
against tiktoken-node
:
Tested on M1 Pro (arm64), 16 GB memory, Node v19.8.1.
(index) | Task Name | Average Time (ms) | Variance (ms) |
---|---|---|---|
0 | 'gpt3-tokenizer' | 27444 | 78653 |
1 | 'gpt-3-encoder' | 16506 | 62548 |
2 | '@dqbd/tiktoken gpt2' | 4194 | 626 |
3 | 'tiktoken-node gpt2' | 3923 | 53 |
4 | '@dqbd/tiktoken text-davinci-003' | 4117 | 58 |
5 | 'tiktoken-node text-davinci-003' | 3840 | 61 |
(Reordered @dqbd/tiktoken
and tiktoken-node
for clarity, PR can be found here: https://github.com/transitive-bullshit/compare-tokenizers/pull/1)
As we can see, tiktoken-node
can be faster than @dqbd/tiktoken-node
, but, as far as I have measured, not 5-6x faster, as claimed (https://github.com/openai/tiktoken/issues/22#issuecomment-1472901919).
I've considered some other cases as well, in case I've missed something:
What if we create a new instance for every iteration (out of 25)?
{
label: '@dqbd/tiktoken gpt2',
encode: (i: string) => {
const tiktokenGpt2 = get_encoding('gpt2')
const result = tiktokenGpt2.encode(i)
tiktokenGpt2.free()
return result
},
},
{
label: 'tiktoken-node gpt2',
encode: (i: string) => {
const tiktokenNode = TiktokenNode.getEncoding('gpt2')
return tiktokenNode.encode(i)
},
}
(index) | Task Name | Average Time (ms) | Variance (ms) |
---|---|---|---|
0 | '@dqbd/tiktoken gpt2' | 227556 | 49932 |
1 | '@dqbd/tiktoken text-davinci-003' | 219893 | 12080 |
2 | 'tiktoken-node gpt2' | 287464 | 241397 |
3 | 'tiktoken-node text-davinci-003' | 303615 | 690935 |
What if we add a much larger fixture in the test suite?
fixtures.push(fixtures[fixtures.length - 1].repeat(100))
(index) | Task Name | Average Time (ms) | Variance (ms) |
---|---|---|---|
0 | '@dqbd/tiktoken gpt2' | 240056 | 33911 |
1 | '@dqbd/tiktoken text-davinci-003' | 238467 | 58049 |
2 | 'tiktoken-node gpt2' | 231459 | 114094 |
3 | 'tiktoken-node text-davinci-003' | 228842 | 126852 |
Update 9/4/2023: Ran the tests again with iterations: 25
Maybe I'm missing something else here? Would it be possible to share and compare benchmarks as well, @ceifa?
Functionality wise, @dqbd/tiktoken
supports more environments (Edge Functions - with ./lite
we even fit within the 1 MB limit) and platforms (browsers - only WASM will be supported). There is some merit in the NAPI approach though, as parallelisation is actually supported in NAPI, thus it might be useful to dig deeper into it.
Hey @dqbd 👋 I made my benchmark in a simple project I was building, so yes, my result can be very innacurate. Indeed your project have much more features and is much more maintained than mine!
Thanks for adding my project on @transitive-bullshit benchmark, so I can try to improve it with things like parallelism, as you said. I think we can work together to delivery the best of the two worlds, since my project can be faster sometimes, and your project can be ran anywhere.
https://github.com/dqbd/tiktoken
For reference, I previously tested a bunch of Node.js tokenizers for accuracy and perf here: https://github.com/transitive-bullshit/compare-tokenizers
cc @dqbd
Thanks! 🙏