dqbd tiktokenizer issues

dqbd / tiktokenizer

Online playground for OpenAPI tokenizers

https://tiktokenizer.vercel.app

MIT License

707 stars 88 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

[wip] see info about tokens, "token-pedia"

#31 darknoon opened 3 weeks ago
1
Add tiptap-based tokenization

#30 dqbd opened 3 weeks ago
1
Can we automatically add input templates for our models?

#28 tcxia opened 3 months ago
0
Add gpt-4o-mini

#27 PeterDaveHello opened 3 months ago
1
[feat] Add docker support

#26 dimaioksha opened 5 months ago
1
Add `gpt-4o` pricing on TokenViewer

#25 kimwonj77 opened 6 months ago
1
Add new `o200k_base` OpenAi encoding for `gpt-4o`

#24 kimwonj77 closed 6 months ago
1
Added and uncommented open source models

#23 philipshurpik opened 6 months ago
1
TOKENS 切割功能

#22 msjsc001 opened 6 months ago
0
Store encoder selection in URL

#21 dqbd closed 8 months ago
1
Support transformers-based tokenizers

#20 darknoon closed 6 months ago
1
changing tokenizer resets the textarea

#19 opyate closed 3 months ago
1
update pricing and differentiate output tokens in price calculation

#18 mahdizaferanchi opened 8 months ago
1
token count is inconsistent with OpenAI tokenizer

#17 GorvGoyl opened 11 months ago
1
Add new OpenAI models

#16 BjoernRave closed 12 months ago
1
How can I get the correct output at the edge?

#15 gablabelle closed 1 year ago
1
Can't run the tiktokenizer from the app folder in next.js

#14 gablabelle opened 1 year ago
2
Mind if the OpenAI Cookbook repo points to your Tiktokenizer app?

#13 ted-at-openai opened 1 year ago
3
Why does the tiktokenizer demo output completely different token numbers than the actual npm package?

#12 Evertt closed 1 year ago
3
[bug] When I use a custom name and then change my mind and try to use one of the default ones, it doesn't update in the textarea below

#11 Evertt opened 1 year ago
0
I see in your demo that I can use a custom name for a message. (How) Is that also possible in OpenAI's API?

#10 Evertt closed 1 year ago
1
Can you add a function to `tiktoken` that automatically adds special characters to chat messages?

#9 Evertt closed 1 year ago
1
Same token same color

#8 IsaacBreen opened 1 year ago
1
Make a Chrome extension for the ChatGPT website that counts number of tokens in input

#7 verhovsky closed 1 year ago
1
Tweak Request: Don't clear prompt textbox when switching models.

#6 JonathanFly closed 1 year ago
4
Add other BPE models

#5 dqbd opened 1 year ago
3
fix: remove the scale effect from button

#4 shadcn closed 1 year ago
2
Use combobox for EncoderSelect

#3 shadcn closed 1 year ago
2
TokenViewer shows "replacement characters" when entering CJK characters

#2 firede closed 1 year ago
1
Quick suggestions

#1 transitive-bullshit closed 1 year ago
1