search

mlc-ai / tokenizers-cpp

Universal cross-platform tokenizers binding to HF and sentencepiece
Apache License 2.0
211 stars 47 forks source link

How to compile in Ubuntu #37

Closed gaokao123 closed 3 weeks ago

gaokao123 commented 3 weeks ago

Error when compiling under ubuntu，my operation is: mkdir build cd build camke .. make

the error is: gaok@lab-System-Product-Name:~/gaokao/3rdLib/tokenizers-cpp/build$ cmake .. -- The C compiler identification is GNU 9.4.0 -- The CXX compiler identification is GNU 9.4.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- system-nameLinux -- Found Doxygen: /usr/bin/doxygen (found version "1.8.17") found components: doxygen missing components: dot -- VERSION: 0.2.00 -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Configuring done -- Generating done -- Build files have been written to: /home/gaok/gaokao/3rdLib/tokenizers-cpp/build gaok@lab-System-Product-Name:~/gaokao/3rdLib/tokenizers-cpp/build$ make [ 2%] Building CXX object CMakeFiles/tokenizer_cpp_objs.dir/src/sentencepiece_tokenizer.cc.o [ 4%] Building CXX object CMakeFiles/tokenizer_cpp_objs.dir/src/huggingface_tokenizer.cc.o [ 6%] Building CXX object CMakeFiles/tokenizer_cpp_objs.dir/src/rwkv_world_tokenizer.cc.o [ 6%] Built target tokenizer_cpp_objs [ 8%] Generating release/libtokenizers_c.a No such file or directory make[2]: [CMakeFiles/tokenizers_c.dir/build.make:71: release/libtokenizers_c.a] Error 1 make[1]: [CMakeFiles/Makefile2:191: CMakeFiles/tokenizers_c.dir/all] Error 2 make: *** [Makefile:156: all] Error 2

Can you give the detailed compilation steps?

gaokao123 commented 3 weeks ago

I got the reason, I didn't install rust. Compile successfully after installing rust

gaokao123 commented 3 weeks ago

must install rust