issues
search
daulet
/
tokenizers
Go bindings for HuggingFace Tokenizer
MIT License
92
stars
23
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Panic when execute tk.Encode
#31
qihuang0
opened
20 hours ago
1
Can padding and/or Truncation be overridden using the library
#30
rmrbytes
opened
4 days ago
0
Can tokenizers.fromPretrained cache directory be specified
#29
rmrbytes
opened
2 weeks ago
3
feat: better error message when tokenizers lib mismatch
#28
daulet
closed
4 weeks ago
0
feat: FromPretrained to load tokenizer directly from HF
#27
berkayersoyy
closed
3 weeks ago
6
WIN11 make build error: it said there is no dictionary target/release/libtokenizers.a
#26
ShowyQuasar88
closed
3 months ago
4
tokenizer.go:190:10: type [1073741824]*_Ctype_char larger than address space
#25
leonardyp
closed
4 weeks ago
3
cannot find -ltokenizers
#24
Doloxetine
closed
3 months ago
1
Update to huggingface/tokenizers v0.20.0
#23
daulet
closed
3 months ago
0
suport for offset mapping?
#22
xuxiaoxia96
closed
3 months ago
1
feat: add option to retrieve offsets from tokenizer
#21
riccardopinosio
closed
3 months ago
2
invalid argument when running example/main.go
#20
lianoid
closed
3 months ago
7
memory issues when using tokenizers
#19
homily707
closed
3 months ago
1
Update to allow for platform dependent libs in CGO
#18
jmoney
closed
5 months ago
4
panic: invalid argument
#17
wuchaowei2012
closed
5 months ago
1
segfault running example main.go
#16
jaybinks
closed
3 months ago
2
Add tokenizers_srcdir_relative build tag to allow static library path
#15
RJKeevil
closed
5 months ago
2
Performance regression
#14
daulet
opened
1 year ago
2
Thread safety issue
#13
RJKeevil
closed
1 year ago
3
Bazel support
#12
daulet
closed
1 year ago
0
example/main.go run error
#11
Crisescode
closed
1 year ago
2
Updated tokenizers for support with Llama models
#10
sam-ulrich1
closed
1 year ago
3
fix cross compilation
#9
jpekmez
closed
11 months ago
1
fix: release-linux
#8
jpekmez
closed
1 year ago
0
fix: panic with cohere tokenizer
#7
jpekmez
closed
1 year ago
0
add encode_batch api and make all encode api support (char) offsets
#6
sunhailin-Leo
closed
1 year ago
0
support more attributes from the Encoding structure
#5
clems4ever
closed
1 year ago
1
libtokenizers.a path should not depend on build source directory
#4
clems4ever
closed
5 months ago
22
add -lm flag to runtime issue
#3
clems4ever
closed
1 year ago
2
encode return tokens
#2
1407010218
closed
1 year ago
1
Added integration tests
#1
michele-sama-dialpad
closed
1 year ago
1