issues
search
openai
/
tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.31k
stars
833
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Provide armv7 wheels (for RaspberryPi)
#86
AlejandroIglesiasCalvo
closed
1 year ago
2
Inconsistency in number of tokens
#85
ahmed-mohamed-sn
closed
1 year ago
1
Estimate total_tokens
#84
loretoparisi
closed
1 year ago
1
Add ruby bindings
#83
arjun810
opened
1 year ago
0
Encoding french document
#82
christopher5106
closed
1 year ago
5
Order by
#81
Yabkonskaya
closed
1 year ago
0
ValueError: Unknown encoding cl100k_base
#80
rembo1992
closed
1 year ago
5
remove duplicated code
#79
GLEF1X
closed
1 year ago
0
Utilize trained huggingface tokenizer with tiktoken
#78
kbmmoran
closed
1 year ago
1
Add a CLI tool
#77
glapa-grossklag
closed
1 year ago
1
tiktoken
#76
eliannaqk
closed
1 year ago
1
Permission denied
#75
realliyifei
closed
1 year ago
6
Add Cargo.lock file
#74
malob
closed
1 year ago
4
MODEL_TO_ENCODING mapping does not work with Azure naming restrictions
#73
chrfla-cognite
closed
1 year ago
9
Patch gpt-35-turbo mapping for Azure naming restrictions
#72
chrfla-cognite
closed
1 year ago
3
Counting tokens from messages (GPT-4)
#71
OTheDev
closed
1 year ago
2
String concatenation / insertion and BPE length
#70
adri1wald
closed
1 year ago
1
Tiktoken dependency : ModuleNotFoundError: No module named 'regex._regex'
#69
Maozerhouni
closed
1 year ago
2
Update the perf image path in readme
#68
Mariatta
closed
1 year ago
0
performance image is broken when viewing the readme from PyPI website
#67
Mariatta
closed
1 year ago
0
Rough token overhead of a message
#66
adri1wald
closed
1 year ago
2
Enable ppc64le wheels
#65
mgiessing
opened
1 year ago
1
There seems to be a problem with tokenizer installation
#64
rahilshah105
closed
1 year ago
2
get_encoding error for gpt2, but other encodings fine
#63
mobilestack
closed
1 year ago
5
No GPT4 support
#62
mermolenko
closed
1 year ago
2
Unable to find ENCODING_CONSTRUCTORS
#61
dheerajiiitv
closed
1 year ago
6
Exporting BPE file to vocab/merges files used by Huggingface tokenizers
#60
colinclement
closed
1 year ago
15
Fix typo, beggining -> beginning
#59
kianmeng
closed
1 year ago
1
tiktoken hangs when getting model encoding.
#58
TravisChristensen
closed
1 year ago
7
how to generate _tiktoken_bg.wasm?
#57
bigbigcoder
closed
1 year ago
1
Add repo description
#56
dandv
closed
1 year ago
1
How to make it support bigscience/bloom ?
#55
hawgjmrd72
closed
1 year ago
3
Build aarch64 wheels under emulation
#54
hauntsaninja
closed
1 year ago
1
Multithreading (Python) vs Parallelism (Rust)
#53
mert-kurttutan
closed
1 year ago
3
Error while installing in Docker dev environment
#52
ershovio
closed
1 year ago
3
unknown encoding
#51
yangwendy
closed
1 year ago
5
Add WASM/JS bindings
#50
dqbd
opened
1 year ago
2
Java Alternative
#49
HarryBouch96
closed
1 year ago
4
Add im_start / im_end in cl100k_base
#48
spolu
opened
1 year ago
4
Thread Panic when decoding token id 100256 and others with cl100k_base tokenizer
#47
minimaxir
closed
1 month ago
3
Not able to Install TikToken : ERROR: Could not find a version that satisfies the requirement tiktoken (from versions: none
#46
clcbot2020
closed
1 year ago
1
gpt-3.5-turbo model support
#45
gobsimpacts0u
closed
1 year ago
2
adds gpt-3.5-turbo and prefix matching to encoding_for_model()
#44
ted-at-openai
closed
1 year ago
2
pyinstaller has some bug that results in improper packaging of tiktoken
#43
bofinbabu
closed
1 year ago
22
Can you add the new ChatGPT API model `gpt-3.5-turbo`
#42
Sea-Snell
closed
1 year ago
4
Does tiktoken support CLIP's tokenizer?
#41
vedantroy
closed
1 year ago
1
WIP: refactor for Java bindings
#40
eisber
opened
1 year ago
2
Follow up question on #https://github.com/openai/tiktoken/issues/38
#39
hatrandev
closed
1 year ago
1
Question on tiktoken vs. GPT2Tokenizer
#38
hatrandev
closed
1 year ago
1
token count differs from the actual token usage from completions API
#37
edwardxwu
closed
1 year ago
2
Previous
Next