issues
search
pkoukk
/
tiktoken-go
go version of tiktoken
MIT License
601
stars
67
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
[Efficiency] Inefficient to create encoders multiple times
#51
omar-scio
opened
2 weeks ago
2
关于支持claude系列模型计量token
#50
FunkyYang
opened
3 weeks ago
0
Any plans on adding Gemini models into collection?
#49
arkadyb
opened
1 month ago
0
feat: include o200k and gpt-4o
#48
jinyuanlu
closed
1 month ago
0
feat: support o200k_base (gpt-4o)
#47
WqyJh
closed
1 month ago
4
新模型的编码
#46
keethebest
closed
1 week ago
0
add o200k_base
#45
MartialBE
closed
1 month ago
1
新增embedding模型
#44
zjy282
closed
1 month ago
1
feat: add support for latest embedding models
#43
matthiasthomas
closed
1 month ago
2
计算结果有误差 我该如何调整
#42
Huxiaoyou97
opened
5 months ago
0
gpt2 support appears missing
#41
tmc
opened
6 months ago
0
tiktoken: add custom constructor
#40
bobheadxi
closed
9 months ago
0
Is there support for function calls tokens
#39
KilianKae
opened
10 months ago
4
sortedTokenBytes doesn't seem necessary
#38
stoneflying
opened
10 months ago
1
Embed the most commonly used vocabularies in the library
#37
shapor
closed
10 months ago
0
Incorrect calculation for Chinese characters
#36
hyp530
closed
10 months ago
1
Fix memory alloc
#35
pkoukk
closed
11 months ago
0
OOM server error
#34
qhenkart
closed
11 months ago
3
Can a Tiktoken instance be shared by multiple go routines?
#33
WqyJh
closed
11 months ago
2
Add 'gpt-3.5-turbo-16k' to NumTokensFromMessages example
#32
zachsmith1
closed
12 months ago
1
Loader
#31
pkoukk
closed
12 months ago
0
Counting Example changed a bit.
#30
nicewook
closed
12 months ago
1
Merge `main` updates into `embed` version
#29
winston-stripe
closed
12 months ago
3
not support for gpt-3.5-turbo-16k-0613
#28
mengmengmengqiang
closed
12 months ago
2
Update README.md
#27
maplessssy
closed
1 year ago
0
Is there any plan to adapt the llama model?
#26
maplessssy
opened
1 year ago
3
enhancement: thread safe for get encoding
#25
pkoukk
closed
1 year ago
0
support all chat model from openai
#24
vaayne
closed
1 year ago
1
tiktoken-go getEmbedding isn't thread-safe
#23
tbiehn
closed
1 year ago
0
add benchmark and benchmark document
#22
pkoukk
closed
1 year ago
0
Add EncoderName() method
#21
bakks
opened
1 year ago
0
Optimize encodeNative()
#20
bakks
closed
1 year ago
1
it‘s result is not ok, compared with openai official tool.
#19
gaoxunyang
closed
1 year ago
0
Update README_zh-hans.md
#18
IT-Talon
closed
1 year ago
0
update readme example
#17
lushenle
closed
1 year ago
0
关于计算误差比较大的问题
#16
ZeroDeng01
closed
1 year ago
1
fix: demo code
#15
gaixianggeng
closed
1 year ago
0
Any benchmark for throughput?
#14
WesleyYue
closed
1 year ago
4
fix: field of name also need to be count
#13
Doublemine
closed
1 year ago
0
EncodingForModel: no encoding for model gpt-3.5-turbo-0301
#12
ethanchancc18
closed
1 year ago
1
Run go mod tidy
#11
philippgille
closed
1 year ago
0
Rename enconding.go to encoding.go
#10
philippgille
closed
1 year ago
0
sync main branch
#9
pkoukk
closed
1 year ago
0
fix allowed and disallowed token usage
#8
pkoukk
closed
1 year ago
0
应当使用 allowedSpecial 以代替 disallowedSpecial??
#7
xgfone
closed
1 year ago
2
计算的token值不准确
#6
chenxi2015
closed
1 year ago
7
feat: embed encoding assets
#5
j178
closed
1 year ago
1
create floder test for compare with openai tiktoken
#4
nasa1024
closed
1 year ago
2
help for count token
#3
137-rick
closed
1 year ago
1
License?
#2
igoose1
closed
1 year ago
1
Next