issues
search
daulet
/
tokenizers
Go bindings for HuggingFace Tokenizer
MIT License
85
stars
23
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
WIN11 make build error: it said there is no dictionary target/release/libtokenizers.a
#26
GinWithoutA
closed
1 month ago
4
tokenizer.go:190:10: type [1073741824]*_Ctype_char larger than address space
#25
leonardyp
opened
1 month ago
3
cannot find -ltokenizers
#24
Doloxetine
closed
1 month ago
1
Update to huggingface/tokenizers v0.20.0
#23
daulet
closed
1 month ago
0
suport for offset mapping?
#22
xuxiaoxia96
closed
1 month ago
1
feat: add option to retrieve offsets from tokenizer
#21
riccardopinosio
closed
1 month ago
2
invalid argument when running example/main.go
#20
lianoid
closed
1 month ago
7
memory issues when using tokenizers
#19
homily707
closed
1 month ago
1
Update to allow for platform dependent libs in CGO
#18
jmoney
closed
3 months ago
4
panic: invalid argument
#17
wuchaowei2012
closed
3 months ago
1
segfault running example main.go
#16
jaybinks
closed
1 month ago
2
Add tokenizers_srcdir_relative build tag to allow static library path
#15
RJKeevil
closed
3 months ago
2
Performance regression
#14
daulet
opened
10 months ago
2
Thread safety issue
#13
RJKeevil
closed
10 months ago
3
Bazel support
#12
daulet
closed
10 months ago
0
example/main.go run error
#11
Crisescode
closed
10 months ago
2
Updated tokenizers for support with Llama models
#10
sam-ulrich1
closed
10 months ago
3
fix cross compilation
#9
jpekmez
closed
9 months ago
1
fix: release-linux
#8
jpekmez
closed
1 year ago
0
fix: panic with cohere tokenizer
#7
jpekmez
closed
1 year ago
0
add encode_batch api and make all encode api support (char) offsets
#6
sunhailin-Leo
closed
10 months ago
0
support more attributes from the Encoding structure
#5
clems4ever
closed
10 months ago
1
libtokenizers.a path should not depend on build source directory
#4
clems4ever
closed
3 months ago
22
add -lm flag to runtime issue
#3
clems4ever
closed
1 year ago
2
encode return tokens
#2
1407010218
closed
1 year ago
1
Added integration tests
#1
michele-sama-dialpad
closed
1 year ago
1