issues
search
zurawiki
/
tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
MIT License
235
stars
45
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Allow users to supply their own BPE tokeniser
#83
xd009642
opened
2 weeks ago
0
Try to fix token counting perf.
#82
xd009642
opened
2 weeks ago
0
Is there a way to cache the bpe when using `num_tokens_from_messages`
#81
xd009642
opened
2 weeks ago
2
hgx
#80
hilop27
closed
3 weeks ago
0
Great to hear! Issues that can be worked on are listed here in the GitHub issues. I'd recommend tackling each issue one at a time. You can comment on each issue that you're interested in working on.
#79
hilop27
closed
3 weeks ago
0
Good points. How are you benchmarking these times today?
#78
hilop27
closed
3 weeks ago
0
chore(deps): update pyo3 requirement from 0.19.2 to 0.22.2
#77
dependabot[bot]
opened
1 month ago
0
chore(deps): update async-openai requirement from 0.14.2 to 0.23.4
#76
dependabot[bot]
opened
1 month ago
0
chore(deps): update pyo3 requirement from 0.19.2 to 0.22.1
#75
dependabot[bot]
closed
1 month ago
1
chore(deps): update pyo3 requirement from 0.19.2 to 0.22.0
#74
dependabot[bot]
closed
1 month ago
1
chore(deps): update async-openai requirement from 0.14.2 to 0.23.3
#73
dependabot[bot]
closed
1 month ago
1
chore(deps): update rustc-hash requirement from 1.1.0 to 2.0.0
#72
dependabot[bot]
opened
2 months ago
0
chore(deps): update async-openai requirement from 0.14.2 to 0.23.2
#71
dependabot[bot]
closed
2 months ago
1
chore(deps): update async-openai requirement from 0.14.2 to 0.23.1
#70
dependabot[bot]
closed
2 months ago
1
chore(deps): bump extractions/setup-just from 1 to 2
#69
dependabot[bot]
opened
2 months ago
0
chore(deps): update base64 requirement from 0.21.5 to 0.22.1
#68
dependabot[bot]
opened
2 months ago
0
chore(deps): update async-openai requirement from 0.14.2 to 0.21.0
#67
dependabot[bot]
closed
2 months ago
1
chore(deps): update pyo3 requirement from 0.19.2 to 0.21.2
#66
dependabot[bot]
closed
1 month ago
1
Remove unnecessary type annotation leading to build errors
#65
xmakro
opened
2 months ago
0
feat: llama3 tokenizer
#64
omarkilani
opened
3 months ago
0
Add support: o200k_base tokenizer.
#63
CaffeeLake
closed
3 months ago
0
Support for other LLMs?
#62
carlca
opened
6 months ago
0
Update README.md
#61
imalsogreg
closed
5 months ago
1
Add `gpt-3.5-turbo-0125`
#60
mikaelsouza
closed
5 months ago
0
Add `gpt-4-0125` and `gpt-3.5-turbo-1106` context sizes
#59
mikaelsouza
closed
7 months ago
1
chore(deps): update pyo3 requirement from 0.19.2 to 0.20.2
#58
dependabot[bot]
closed
2 months ago
1
chore(deps): update async-openai requirement from 0.14.2 to 0.18.0
#57
dependabot[bot]
closed
2 months ago
1
async-openai 0.18 support
#56
ChristopherJMiller
opened
7 months ago
1
chore(deps): update pyo3 requirement from 0.19.2 to 0.20.1
#55
dependabot[bot]
closed
7 months ago
1
chore(deps): update fancy-regex requirement from 0.12.0 to 0.13.0
#54
dependabot[bot]
opened
8 months ago
0
Add tokenizer prefixes for fine-tuned models
#53
jbgriesner
closed
8 months ago
1
chore(deps): update async-openai requirement from 0.14.2 to 0.17.1
#52
dependabot[bot]
closed
7 months ago
1
chore(deps): update async-openai requirement from 0.14.2 to 0.16.3
#51
dependabot[bot]
closed
9 months ago
1
Update to async-openai 0.16.2
#50
Dreaming-Codes
opened
9 months ago
3
Add gpt-4-1106-preview model
#49
JosephTLyons
closed
9 months ago
2
chore(deps): update async-openai requirement from 0.14.2 to 0.16.2
#48
dependabot[bot]
closed
9 months ago
1
chore(deps): update fancy-regex requirement from 0.11.0 to 0.12.0
#47
dependabot[bot]
closed
8 months ago
1
Add `gpt-4-1106-preview`
#46
JosephTLyons
closed
9 months ago
3
Make encode token generic, accepting usize and u32
#45
mokeyish
opened
9 months ago
1
chore(deps): update async-openai requirement from 0.14.2 to 0.15.0
#44
dependabot[bot]
closed
9 months ago
1
Expose _decode_native methods
#43
zurawiki
closed
10 months ago
0
chore(deps): update pyo3 requirement from 0.19.2 to 0.20.0
#42
dependabot[bot]
closed
8 months ago
1
return correct context length for `text-embedding-ada-002`
#41
ursachec
closed
10 months ago
3
Token counts are incorrect when request includes function messages
#40
kylewlacy
opened
10 months ago
1
BPE memory leak
#39
venual
opened
11 months ago
7
lint: Fix unnecessary raw string
#38
zurawiki
closed
11 months ago
0
Don't arbitrarily enforce versions please
#37
aristotaloss
closed
11 months ago
1
feat: add multi-language bindings for tiktoken using polywrap wasm wrapper
#36
Niraj-Kamdar
opened
11 months ago
0
chore(deps): update bstr requirement from 0.2.17 to 1.6.2
#35
dependabot[bot]
closed
11 months ago
1
chore(deps): bump actions/checkout from 3 to 4
#34
dependabot[bot]
closed
11 months ago
0
Next