issues
search
mlc-ai
/
tokenizers-cpp
Universal cross-platform tokenizers binding to HF and sentencepiece
Apache License 2.0
210
stars
46
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
[CMake] Support disable SentencePiece tokenizer
#38
MasterJH5574
closed
5 days ago
0
How to compile in Ubuntu
#37
gaokao123
closed
3 weeks ago
2
[Rust] Bump HuggingFace tokenizer version
#36
MasterJH5574
closed
1 month ago
0
tokenizer for triton inference server
#35
geraldstanje
opened
1 month ago
0
Remove incorrect final for non-virtual func
#34
vinx13
closed
1 month ago
0
Added EncodeBatch interface
#33
vinx13
closed
1 month ago
0
Fail to import in Node.js
#32
zcbenz
opened
1 month ago
0
Problem resolving some symbols when using the library in an Android C++ project (I am compiling using ndk)
#31
cs-jlopezr
opened
2 months ago
2
conflict with torch
#30
zhangyuhanjc
closed
2 months ago
1
compile failed
#29
chenpei2
closed
2 months ago
1
Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI
#28
yuxuanchiadm
closed
2 months ago
1
Allow parameters related to special tokens for HFTokenizer
#27
Abhishek8394
closed
2 months ago
0
Allow hugginface tokenizer to pass arguments for add/skip special tokens
#26
Abhishek8394
closed
2 months ago
1
[Web] Expose getVocabSize and idToToken to web, bump version to 0.1.3
#25
CharlieFRuan
closed
3 months ago
0
Can't support cross build on Linux
#24
SakuragiJump
closed
2 months ago
5
is there any plan to support Tiktoken?
#23
Jasonsey
opened
6 months ago
2
Add support for querying vocabulary from tokenizer
#22
Ubospica
closed
6 months ago
5
Error when compiling tokenizers for AMD with HIP C++ compiler (hipcc)
#21
goliaro
closed
6 months ago
1
Update build.sh to make web tokenizer work with chrome extension
#20
manuongithub
closed
7 months ago
2
fix rwkv world tokenzier
#19
BBuf
closed
7 months ago
0
Update to reduce memory realloc
#18
tqchen
closed
8 months ago
0
How to free encode ids without destructing tokenizer?
#17
ShukantPal
closed
2 months ago
8
[FIX] Remove boost dependency from submodule msgpack
#16
Hzfengsy
closed
10 months ago
0
msgpack as 3rdparty library instead of fetch_context
#15
BBuf
closed
10 months ago
1
add rwkv world tokenizer
#14
BBuf
closed
10 months ago
6
If the model hasn't tokenizer.json file, what should I do?
#13
wolf-li
closed
2 months ago
2
Update rust to support latest tokenizers
#11
tqchen
closed
10 months ago
0
Compiler error (Rust)
#10
jklaise
closed
10 months ago
3
undefined symbol: open64 when run build.sh in web dir
#9
burke-up
closed
10 months ago
1
An error occurred in the compilation
#8
Liu-xiandong
closed
11 months ago
1
SentencePiece Build Error - ld: error: undefined symbol: __android_log_write
#7
zjc664656505
closed
11 months ago
6
Update CMakeLists.txt
#6
luiyen
closed
11 months ago
1
Fix windows build by linking against ntdll
#5
junrushao
closed
1 year ago
0
Fix readme typo
#4
erjanmx
closed
1 year ago
1
Add npm package
#3
tqchen
closed
1 year ago
0
Update CMakeLists.txt
#2
songkq
closed
1 year ago
1
add_library INTERFACE library requires no source arguments.
#1
songkq
closed
1 year ago
12