issues
search
openai
/
tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.48k
stars
856
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Python memory usage
#242
logan-markewich
closed
9 months ago
6
Gitkraken
#241
Kedzia00
closed
10 months ago
0
Good off day
#240
Kedzia00
closed
10 months ago
0
Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)
#239
l0rinc
opened
10 months ago
4
Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)
#238
blaney83
opened
10 months ago
1
0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings
#237
l0rinc
opened
10 months ago
0
new method to truncate after N-tokens
#236
aleks-sch
closed
10 months ago
2
tiktoken not found even though its installed.
#235
markcam1
closed
10 months ago
1
1) Optimize regular expressions used for splitting by ~20%
#234
l0rinc
closed
9 months ago
0
Is the model link invalid？
#233
igorwang
closed
10 months ago
0
Is there a way for tiktoken to interoperate better with offline AI software?
#232
ParetoOptimalDev
opened
11 months ago
3
Pickling tokenizer fails due to builtins.CoreBPE
#231
jerheff
closed
9 months ago
4
Add support for checking hash of downloaded files before use.
#230
mdwelsh
closed
10 months ago
2
Moi
#229
4579610615
closed
11 months ago
0
Tiktoken
#228
Kedzia00
closed
11 months ago
2
F-REQ: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer
#227
Emasoft
opened
11 months ago
2
Update registry.py - Remove duplicate lines
#226
jmishra01
closed
10 months ago
1
What Does "CL" Stand For in CL100K?
#225
jdmccaffrey
closed
11 months ago
1
How to install 0.3.3?
#224
AZmisc
closed
11 months ago
1
Please add new mode gpt-3.5-turbo-instruct
#223
kensenjohn
closed
9 months ago
1
No response when more than 32k tokens requested
#221
fanpeeps
closed
11 months ago
1
encoding_for_model("gpt-4")
#220
fanpeeps
closed
11 months ago
1
Use a fresh tempdir for data-gym-cache to avoid "Permission denied"
#219
wookayin
closed
11 months ago
1
Plugins found: ['tiktoken_ext.openai_public']
#218
voghoei
closed
1 month ago
15
Update __init__.py
#216
ansumanswain
closed
1 year ago
0
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects
#215
2anirban
closed
12 months ago
8
Add gpt-4-1106-preview model
#214
JosephTLyons
closed
9 months ago
0
replace requests with httpx
#213
singingwolfboy
closed
11 months ago
1
My server cannot connect to openaipublic.blob.core.windows.net. Where can I download the cl100k_base file and how can I cache it?
#212
erjiguan
closed
1 year ago
1
Connection reset by peer when the server get bpe file
#211
leantli
closed
1 year ago
1
Using tiktoken with async code (python)
#210
sometastycake
closed
1 year ago
3
tiktoken does not work with blobfile==2.1.0
#209
Praful932
closed
1 year ago
2
Fix blobfile dependency
#208
Praful932
closed
1 year ago
1
Nuitka compiler, unknown encoding "ck100_base"
#207
aspoofer0224
closed
1 year ago
1
Create MEXICO
#206
ABDULLAH9119CVV
closed
1 year ago
0
Can't install tiktoken in Python 3.12
#205
pamelafox
closed
11 months ago
7
اخ
#204
1ysc7
closed
1 year ago
0
Update README.md to clarify context of models
#203
logankilpatrick
closed
10 months ago
0
Tiktoken failed to decode Math symbols
#202
ahmedmoorsy
closed
1 year ago
1
Unable to install tiktoken in ubuntu 18.04 server
#201
nagendra-rqsr
closed
1 year ago
2
Add Python 3.12 build wheel
#200
rinarakaki
closed
11 months ago
12
BPE Memory leak
#199
venual
closed
1 year ago
1
tiktoken error on AWS lambda
#198
vicky141998
opened
1 year ago
5
tiktoken fails with Rust 1.60 since memchr 2.6.3 got released
#197
jhgoebbert
closed
11 months ago
3
Tiktoken's number of tokens does not match the number of tokens perceived by the gpt-3.5-turbo API
#196
viswavi
closed
11 months ago
1
Very slow for inputs like "a" * 100000
#195
vvolhejn
opened
1 year ago
10
Running import tiktoken results in ImportError
#194
carlsmedstad
closed
1 year ago
2
[haystack][Backward compatibility] MODEL_TO_ENCODING instead of _MODEL_TO_ENCODING, similarly for MODEL_PREFIX_TO_ENCODING
#193
cppxaxa
closed
1 year ago
5
Update on the tiktoken tokenizer module?
#192
HiThere0175
closed
1 year ago
3
add __version__ attribute
#191
rasbt
closed
1 year ago
1
Previous
Next