issues
search
openai
/
tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.76k
stars
801
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
The calculation results different with Tokenizer
#120
mysaber
closed
1 year ago
4
Get_encoding failures in aws lambda execution
#119
jhb96
closed
1 year ago
1
Remove unused code
#118
laran
closed
1 year ago
2
Build wheel for pyodide, take 2
#117
wch
closed
1 year ago
1
Requests should be an optional dependency
#116
rhuddleston
closed
1 year ago
1
tiktoken should include function num_tokens_from_messages()
#115
rmilkowski
closed
1 year ago
1
Not enough values error in tiktoken.get_encoding
#114
Ayan-Bandyopadhyay
closed
1 year ago
3
New followers for TikTok
#113
Rombusek
closed
1 year ago
0
Is this not really similar to the tokenizer ChatGPT uses? (For counting tokens at least)
#112
NPap0
closed
1 year ago
2
Enhancing CJK Performance: Accelerating Training and Inference by 2-3 Times
#111
golbin
closed
1 year ago
1
Since openai does not provide a Java version of the method for calculating gpt-3.5 tokens, can I use Java to run Python code?
#110
TreeFireMen
closed
1 year ago
1
Create a tiktoken.max_tokens_for_model() function?
#109
endolith
closed
1 year ago
2
Allow older versions of dependencies fancy-regex and bstr.
#108
petterreinholdtsen
opened
1 year ago
1
cant install
#107
valentMeng
closed
1 year ago
0
most likely due to a circular import
#106
minkai95
closed
1 year ago
3
modify cache_dir permission on creation: RWX for all users
#105
HUGHNew
closed
9 months ago
1
cannot install tiktoken using pip
#104
Pusse-01
closed
1 year ago
1
Fix missing line numbers in stack traces
#103
christopherhesse
opened
1 year ago
0
The tokenized results are not as expected when I added special tokens
#102
tianxie-9
closed
1 year ago
7
Refactored 'test_simple()'.
#101
Ramseths
closed
1 year ago
1
encoding.decode() function does not allow filtering out special tokens
#100
sproocht
closed
1 year ago
3
Tiktoken picking shorter tokens resulting in more tokens
#99
bluescreen10
closed
1 year ago
2
FAQ (please read before opening a new issue)
#98
hauntsaninja
opened
1 year ago
0
Unofficial bindings / ports in other languages
#97
hauntsaninja
opened
1 year ago
11
Update platform.openai.com/tokenizer ?
#96
NPap0
closed
1 year ago
1
Can tiktoken be utilized with all GPT models?
#95
yachty66
closed
1 year ago
1
Add support for Pyodide
#94
Emasoft
closed
1 year ago
6
fix error message
#93
jonathanagustin
closed
1 year ago
1
License for tokenizer models are unclear
#92
chris-ha458
closed
1 year ago
2
Installing paddleocr>=2.6.1.0, TomlError happend
#90
luoshenfu
closed
1 year ago
2
Installation Issue (TomlError)
#89
ManishSihag
closed
1 year ago
1
add cargo lock for reproducibility
#88
happysalada
closed
9 months ago
5
Error: pyo3_runtime.PanicException assertion failed: encoder.len() == decoder.len()
#87
osjason
closed
1 year ago
4
Provide armv7 wheels (for RaspberryPi)
#86
AlejandroIglesiasCalvo
closed
1 year ago
2
Inconsistency in number of tokens
#85
ahmed-mohamed-sn
closed
1 year ago
1
Estimate total_tokens
#84
loretoparisi
closed
1 year ago
1
Add ruby bindings
#83
arjun810
opened
1 year ago
0
Encoding french document
#82
christopher5106
closed
1 year ago
5
Order by
#81
Yabkonskaya
closed
1 year ago
0
ValueError: Unknown encoding cl100k_base
#80
rembo1992
closed
1 year ago
5
remove duplicated code
#79
GLEF1X
closed
1 year ago
0
Utilize trained huggingface tokenizer with tiktoken
#78
kbmmoran
closed
1 year ago
1
Add a CLI tool
#77
glapa-grossklag
closed
1 year ago
1
tiktoken
#76
eliannaqk
closed
1 year ago
1
Permission denied
#75
realliyifei
closed
1 year ago
6
Add Cargo.lock file
#74
malob
closed
1 year ago
4
MODEL_TO_ENCODING mapping does not work with Azure naming restrictions
#73
chrfla-cognite
closed
1 year ago
9
Patch gpt-35-turbo mapping for Azure naming restrictions
#72
chrfla-cognite
closed
1 year ago
3
Counting tokens from messages (GPT-4)
#71
OTheDev
closed
1 year ago
2
String concatenation / insertion and BPE length
#70
adri1wald
closed
1 year ago
1
Previous
Next