issues
search
openai
/
tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.17k
stars
751
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Create MEXICO
#206
ABDULLAH9119CVV
closed
9 months ago
0
Can't install tiktoken in Python 3.12
#205
pamelafox
closed
7 months ago
7
اخ
#204
1ysc7
closed
9 months ago
0
Update README.md to clarify context of models
#203
logankilpatrick
closed
5 months ago
0
Tiktoken failed to decode Math symbols
#202
ahmedmoorsy
closed
9 months ago
1
Unable to install tiktoken in ubuntu 18.04 server
#201
nagendra-rqsr
closed
9 months ago
2
Add Python 3.12 build wheel
#200
rinarakaki
closed
7 months ago
12
BPE Memory leak
#199
venual
closed
9 months ago
1
tiktoken error on AWS lambda
#198
vicky141998
opened
10 months ago
5
tiktoken fails with Rust 1.60 since memchr 2.6.3 got released
#197
jhgoebbert
closed
7 months ago
3
Tiktoken's number of tokens does not match the number of tokens perceived by the gpt-3.5-turbo API
#196
viswavi
closed
7 months ago
1
Very slow for inputs like "a" * 100000
#195
vvolhejn
opened
10 months ago
10
Running import tiktoken results in ImportError
#194
carlsmedstad
closed
10 months ago
2
[haystack][Backward compatibility] MODEL_TO_ENCODING instead of _MODEL_TO_ENCODING, similarly for MODEL_PREFIX_TO_ENCODING
#193
cppxaxa
closed
10 months ago
5
Update on the tiktoken tokenizer module?
#192
HiThere0175
closed
10 months ago
3
add __version__ attribute
#191
rasbt
closed
10 months ago
1
Adding a __version__ attribute
#190
rasbt
opened
10 months ago
4
d
#189
belpolzkzkoz
closed
10 months ago
0
Why does "llo" result in two tokens and not in one?
#188
Kamilcuk
closed
10 months ago
1
What is the difference between gpt2 encoding and r50k_base?
#187
Mypathissional
closed
10 months ago
1
Replace `<|endoftext|>` with `ENDOFTEXT`
#186
alvarobartt
closed
10 months ago
1
Improve `tiktoken.encoding_for_model` with better option
#185
xhluca
closed
5 months ago
1
from tiktoken import _tiktoken is causing circular dependency on AWS lambda
#184
akshat-g
closed
5 months ago
11
How to forbid tokens for specific words only?
#183
Clement-Lelievre
closed
10 months ago
1
ChatGPT
#182
jschwartz996
closed
10 months ago
0
Make `Encoding` serializable
#181
mariosasko
closed
5 months ago
3
Support newer completion models in `encoding_for_model`
#180
rfriel
closed
10 months ago
1
Please commit your Cargo.lock file in the repository
#179
charles-dyfis-net
opened
11 months ago
3
Allow option to load in encodings ref from local file path instead of external call
#178
nanditamohan
closed
7 months ago
4
Numbers + whitespaces are not tokenized properly
#177
bzwheeler
closed
11 months ago
1
Non-decodable tokens
#176
microsoftbuild
closed
11 months ago
2
TikToken.EncodingForModel("gpt-3.5-turbo") hangs when context is a WPF appplication.
#175
felipedrumond
closed
11 months ago
2
I added words from other languages, similar to Chinese, which caused the segmentation to fail properly, even though the word was in the vocabulary, the word was still split into multiple bytes in a sentence
#174
WUHU-G
closed
10 months ago
3
For Android
#173
leotorenzo6
closed
11 months ago
1
compilation error during installation via pip
#172
Darskiy
closed
11 months ago
5
Updated Readme
#171
skfrost19
closed
7 months ago
1
feat: Add option TIKTOKEN_FORCE_CACHE
#170
Kontrolix
opened
11 months ago
0
tiktoken_ext.__pycache__ Issue On Amazone Lambda
#169
FarzamHejaziK
opened
12 months ago
1
ValueError: Unknown encoding cl100k_base error when program is compiled
#168
tomhayw
closed
12 months ago
1
✨ feat: Split code rust, for run as python lib and rust lib
#167
Miuler
opened
1 year ago
5
Deprecation of older models
#166
AlexanderMorou
closed
1 year ago
3
Token
#165
leotorenzo6
closed
1 year ago
0
Pricing
#164
macarneiro1
closed
1 year ago
1
Unknown encoding cl100k_base when building program using pyinstaller
#163
Klyser8
closed
1 year ago
1
BOOL-1940: Refactor code to work with Python 3.7
#162
Mainstay-Noah-Huppert
closed
1 year ago
0
Fatal Python error: Segmentation fault from `tiktoken/core.py`
#161
yuvalshi0
closed
12 months ago
3
Calculate token consumption for OpenAI function definitions
#160
Pwuts
closed
1 year ago
1
Inconsistent "\n\n" encoding
#159
vince62s
closed
1 year ago
4
大量tiktok真人实号
#158
Tbagbast
closed
1 year ago
0
Feature Request: return dollar amount for OpenAI request response
#157
dhruv-anand-aintech
closed
1 year ago
1
Previous
Next