search

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
https://gpt-tokenizer.dev
MIT License
434 stars 35 forks source link

feat: significantly improve performance, memory usage & initialization time #50

Closed niieani closed 2 weeks ago

niieani commented 2 weeks ago 
│ Package        │ Version │ Init   │ Init  │ Init    │ Encode   │ Decode   │ Memory   │ Mem   │
│                │         │ (ms)   │ Mem ^ │ Mem RSS │ Avg (ms) │ Avg (ms) │          │ Leak? │
│ gpt-tokenizer  │ 2.2.3   │ 253.04 │ 45.83 │ 150.42  │ 0.0208   │ 0.0033   │ 0.21     │ No    │
├────────────────┼─────────┼────────┼───────┼─────────┼──────────┼──────────┼──────────┼───────┤
│ gpt-tokenizer  │ next    │ 44.58  │ 9.65  │ 35.76   │ 0.0102   │ 0.0025   │ 0.25     │ No    │
└────────────────┴─────────┴────────┴───────┴─────────┴──────────┴──────────┴──────────┴───────┘
github-actions[bot] commented 2 weeks ago

:tada: This PR is included in version 2.3.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: