issues
search
zurawiki
/
tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240
stars
47
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
chore(deps): update async-openai requirement from 0.13.0 to 0.14.0
#33
dependabot[bot]
closed
1 year ago
1
Derive Debug for CoreBPE struct
#32
tristan-jl
closed
1 year ago
1
Update to async-openai 0.13.0
#31
Dreaming-Codes
closed
1 year ago
1
[Question] Difference between `encode_with_special_tokens` and `encode_ordinary`
#30
kojix2
opened
1 year ago
0
tiktoken-c and tiktoken-cr
#29
kojix2
closed
1 year ago
0
minor change
#28
kojix2
closed
1 year ago
1
Expose `_decode_native`?
#27
Palmik
closed
10 months ago
4
Update async-openai to 0.12.0
#26
DaviRain-Su
closed
1 year ago
1
chore(deps): update async-openai requirement from 0.11.1 to 0.12.0
#25
dependabot[bot]
closed
1 year ago
1
test: Add unicode test cases
#24
zurawiki
closed
1 year ago
0
Incomplete utf-8 byte sequence from index 0
#23
BohuTANG
closed
1 year ago
1
Add README.md to package
#22
fiji-flo
closed
1 year ago
1
chore(deps): update async-openai requirement from 0.10.3 to 0.11.0
#21
dependabot[bot]
closed
1 year ago
0
Bench
#20
zurawiki
closed
1 year ago
0
chore(deps): update pyo3 requirement from 0.18.3 to 0.19.0
#19
dependabot[bot]
closed
1 year ago
0
Improved interface for split_by_token
#18
jackbackes
closed
1 year ago
2
Add ability to tokenize a string and return the decoded tokens using the correct BPE model
#17
jackbackes
closed
1 year ago
2
Use case: splitting text into tokens.
#16
jackbackes
closed
1 year ago
0
feat: make async-openai optional
#15
j178
closed
1 year ago
0
fix: expose num_tokens_from_messages and ensure its implementation is consistent with OpenAI
#14
j178
closed
1 year ago
0
Make the crate independent of async_openai
#13
thedenisnikulin
closed
1 year ago
1
chore: Update dependencies and versions
#12
zurawiki
closed
1 year ago
0
chore(deps): update async-openai requirement from 0.9.5 to 0.10.0
#11
dependabot[bot]
closed
1 year ago
0
Any chance of isolating some dependencies with features?
#10
IAPark
closed
1 year ago
3
feat: Add gpt-4 models mapping and tests.
#9
zurawiki
closed
1 year ago
0
Missing gpt2 encoder
#8
ceifa
opened
1 year ago
1
How could you add parallelism to make the encoding faster?
#7
smahm006
opened
1 year ago
6
Do not build python bindings by default
#6
kelvich
closed
1 year ago
3
use tiktoken encode throw error.
#5
misssonder
closed
1 year ago
1
Vendor out the tiktoken code
#4
zurawiki
closed
1 year ago
0
Refactor tokenization and special tokens
#3
zurawiki
closed
1 year ago
0
Implement cl100k tokenizer
#2
zurawiki
closed
1 year ago
0
cl100k usage
#1
mkualquiera
closed
1 year ago
1
Previous