niieani gpt-tokenizer issues

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.

https://gpt-tokenizer.dev

MIT License

434 stars 35 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

chore(deps): update github/codeql-action action to v3

#55 renovate[bot] opened 4 days ago
0
chore(deps): update actions/setup-node action to v4

#54 renovate[bot] opened 4 days ago
0
chore(deps): update dependency @niieani/scaffold to ^1.7.34

#53 renovate[bot] opened 4 days ago
1
feat: performance optimizations

#52 niieani closed 1 week ago
1
chore(deps): lock file maintenance

#51 renovate[bot] opened 1 week ago
1
feat: significantly improve performance, memory usage & initialization time

#50 niieani closed 2 weeks ago
1
[Bug] Cannot upgrade to TS 5.6x due to build errors

#49 corinagum closed 2 weeks ago
7
empty chat will return 3 tokens

#48 BennyAlex closed 2 weeks ago
1
encodeChat: lineToEncode.match is not a function

#47 BennyAlex closed 2 weeks ago
3
Gpt 4o mini support?

#45 semanticist21 closed 2 months ago
1
fix(deps): update dependency rfc4648 to ^1.5.3

#44 renovate[bot] closed 2 weeks ago
2
chore: setup o200k_base tokenizer

#43 mbukeRepo closed 2 months ago
4
Support New o200k_base

#42 RedwindA closed 2 months ago
1
Picture tokens

#41 arthurwolf opened 8 months ago
3
vercel/pkg Error with gpt-tokenizer

#40 Ranork closed 2 weeks ago
1
the `isWithinTokenLimit` may be broken

#39 zhouhan760503 closed 9 months ago
1
Is the algorithm of this repo consistent with openai tiktoken?

#38 Zwe1 closed 2 months ago
1
Error: Model name must be provided either during initialization or passed in to the method

#37 thdoan closed 11 months ago
2
ChatMessage[] TypeScript warning

#36 thdoan closed 2 months ago
3
Huge memory consumsation of isWithinTokenLimit

#35 aminsol closed 2 weeks ago
6
How to set the model in encode / encode generator like encodeChat or encodeChatGenerator?

#34 MiniSuperDev closed 2 months ago
1
chore(deps): update actions/checkout action to v4

#33 renovate[bot] closed 2 months ago
1
Different calculating result between gpt-tokenizer and openai

#32 Penggeor closed 1 year ago
1
chore(deps): update dependency tsx to ^3.12.10 - autoclosed

#31 renovate[bot] closed 2 weeks ago
2
Error in NextJS edge function: Attempted import error: './index.js' does not contain a default export (imported as 'cjs')

#30 gablabelle closed 2 months ago
2
ReferenceError: Property 'TextDecoder' doesn't exist, js engine: hermes

#28 YunySun closed 2 months ago
1
Error: Invariant: Method expects to have requestAsyncStorage, none available

#27 chenqilin70 closed 2 months ago
0
isWithinTokenLimit fails due to lack of model being provided

#26 emfastic closed 2 months ago
7
chore(deps): update dependency @niieani/scaffold to ^1.7.32 - autoclosed

#25 renovate[bot] closed 2 weeks ago
2
chore(deps): update yarn to v3.8.5 - autoclosed

#24 renovate[bot] closed 2 weeks ago
2
chore(deps): update yarn to v3.5.1 - autoclosed

#23 renovate[bot] closed 2 weeks ago
2
Improving documentation about the use of api vs importing function directly

#22 nahumzs opened 1 year ago
2
Support function calls?

#21 seoker opened 1 year ago
3
Tokens are slightly different from OpenAI Tokenizer

#20 nilsreichardt closed 1 year ago
1
Add support for `gpt-3.5-turbo-0613` and `gpt-3.5-turbo-16k-0613`

#19 nilsreichardt closed 2 months ago
2
avoid expensive initialization

#18 mimoo closed 2 weeks ago
8
最近官方更新了GPT3.5-16K的模型，请问计算方式跟gpt-3.5-turbo是一样的吗

#17 xiaoqixiaoqi1113 closed 2 months ago
1
This module is not ready for CJK characters

#16 mashihua closed 2 months ago
3
encodeChatGenerator undefined?

#15 congqi09 closed 12 months ago
16
Dependency Dashboard

#14 renovate[bot] opened 1 year ago
0
Add gpt-4-32k to mappings

#13 wesbos closed 1 year ago
1
Loading with script tags doesn't work

#12 baobabKoodaa closed 1 year ago
1
comparison to other tokenizers

#11 transitive-bullshit closed 1 year ago
2
How can I get prompts_tokens?

#10 magelikescoke closed 1 year ago
3
gpt-tokenizer@2.0.0 token count is smaller than ChatGPT api response

#9 Webifi closed 1 year ago
1
feat: complete rewrite to support different models

#8 niieani closed 1 year ago
1
CDN version is broken

#7 alejandro5042 closed 1 year ago
2
Calculated tokens much higher than actual

#6 Qarj closed 1 year ago
10
Which encoding is this using?

#5 ricardomatias closed 1 year ago
3
Import Issue on codebases with both module and common js files

#4 gbrlmtrz closed 1 year ago
4