openai tiktoken issues - Githubissues

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.06k stars 749 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Simplify byte_pair_merge

#255 hauntsaninja closed 4 months ago
1
69898163

#254 hebosho911 closed 4 months ago
0
Inline custom mapping function in _byte_pair_merge

#253 hauntsaninja closed 4 months ago
0
Avoid calling byte_pair_encode for existing tokens

#252 hauntsaninja closed 4 months ago
0
Store tokens in u32 instead of usize

#251 hauntsaninja closed 4 months ago
0
Enhancement: Add convenience token-counting functions to this package

#250 pamelafox opened 5 months ago
4
Are new line characters separate tokens?

#249 GlassBeaver closed 5 months ago
1
Adds caching to get_encoding to avoid repeatedly constructing Encodings

#248 tal7aouy closed 5 months ago
1
added two new embedding model's encoding

#247 Praneet460 closed 4 months ago
11
HBushIA

#246 hebosho911 closed 5 months ago
0
Panic (stack overflow) when encoding a certain string

#245 Crazytieguy opened 5 months ago
3
Junhyun/add upstage solar

#244 jhpark-upstage closed 5 months ago
0
junhyun/add_upstage_solar

#243 jhpark-upstage closed 5 months ago
0
Python memory usage

#242 logan-markewich closed 5 months ago
6
Gitkraken

#241 Kedzia00 closed 5 months ago
0
Good off day

#240 Kedzia00 closed 5 months ago
0
Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)

#239 paplorinc opened 5 months ago
4
Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)

#238 blaney83 opened 5 months ago
1
0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings

#237 paplorinc opened 6 months ago
0
new method to truncate after N-tokens

#236 aleks-sch closed 5 months ago
2
tiktoken not found even though its installed.

#235 markcam1 closed 6 months ago
1
1) Optimize regular expressions used for splitting by ~20%

#234 paplorinc closed 4 months ago
0
Is the model link invalid？

#233 igorwang closed 5 months ago
0
Is there a way for tiktoken to interoperate better with offline AI software?

#232 ParetoOptimalDev opened 6 months ago
3
Pickling tokenizer fails due to builtins.CoreBPE

#231 jerheff closed 4 months ago
4
Add support for checking hash of downloaded files before use.

#230 mdwelsh closed 5 months ago
2
Moi

#229 4579610615 closed 6 months ago
0
Tiktoken

#228 Kedzia00 closed 6 months ago
2
F-REQ: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer

#227 Emasoft opened 7 months ago
1
Update registry.py - Remove duplicate lines

#226 jmishra01 closed 5 months ago
1
What Does "CL" Stand For in CL100K?

#225 jdmccaffrey closed 6 months ago
1
How to install 0.3.3?

#224 AZmisc closed 7 months ago
1
Please add new mode gpt-3.5-turbo-instruct

#223 kensenjohn closed 4 months ago
1
No response when more than 32k tokens requested

#221 fanpeeps closed 7 months ago
1
encoding_for_model("gpt-4")

#220 fanpeeps closed 7 months ago
1
Use a fresh tempdir for data-gym-cache to avoid "Permission denied"

#219 wookayin closed 7 months ago
1
Plugins found: ['tiktoken_ext.openai_public']

#218 voghoei opened 7 months ago
13
Update __init__.py

#216 ansumanswain closed 7 months ago
0
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

#215 2anirban closed 7 months ago
8
Add gpt-4-1106-preview model

#214 JosephTLyons closed 4 months ago
0
replace requests with httpx

#213 singingwolfboy closed 7 months ago
1
My server cannot connect to openaipublic.blob.core.windows.net. Where can I download the cl100k_base file and how can I cache it?

#212 erjiguan closed 8 months ago
1
Connection reset by peer when the server get bpe file

#211 leantli closed 8 months ago
1
Using tiktoken with async code (python)

#210 sometastycake closed 8 months ago
3
tiktoken does not work with blobfile==2.1.0

#209 Praful932 closed 8 months ago
2
Fix blobfile dependency

#208 Praful932 closed 8 months ago
1
Nuitka compiler, unknown encoding "ck100_base"

#207 aspoofer0224 closed 8 months ago
1
Create MEXICO

#206 ABDULLAH9119CVV closed 9 months ago
0
Can't install tiktoken in Python 3.12

#205 pamelafox closed 7 months ago
6
اخ

#204 1ysc7 closed 9 months ago
0

Previous Next