search

mlc-ai / tokenizers-cpp

Universal cross-platform tokenizers binding to HF and sentencepiece
Apache License 2.0
211 stars 47 forks source link

compile failed #29

Closed chenpei2 closed 2 months ago

chenpei2 commented 2 months ago

Command: mkdir build cd build cmake .. make

root@k8s-master-102-devel:/code/tokenizers-cpp/build# make [ 2%] Building CXX object CMakeFiles/tokenizer_cpp_objs.dir/src/sentencepiece_tokenizer.cc.o [ 4%] Building CXX object CMakeFiles/tokenizer_cpp_objs.dir/src/huggingface_tokenizer.cc.o [ 6%] Building CXX object CMakeFiles/tokenizer_cpp_objs.dir/src/rwkv_world_tokenizer.cc.o [ 6%] Built target tokenizer_cpp_objs [ 8%] Generating release/libtokenizers_c.a no such file or directory make[2]: [CMakeFiles/tokenizers_c.dir/build.make:71: release/libtokenizers_c.a] Error 1 make[1]: [CMakeFiles/Makefile2:191: CMakeFiles/tokenizers_c.dir/all] Error 2 make: *** [Makefile:156: all] Error 2

chenpei2 commented 2 months ago

problem solved after installing rust. curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh