search

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.5k stars 513 forks source link

Is the c++ example working? #195

Closed nicolaskodak closed 5 years ago

nicolaskodak commented 5 years ago

Hi,

have spent several hours but can't make the C++ example on page to work, so decide to post it here.

c++ code (example.cpp)

#include "lm/model.hh"
#include <iostream>
#include <string>
int main() {
  using namespace lm::ngram;
  Model model("file.arpa");
  State state(model.BeginSentenceState()), out_state;

  const Vocabulary &vocab = model.GetVocabulary();
  std::string word;
  while (std::cin >> word) {
    std::cout << model.Score(state, vocab.Index(word), out_state) << '\n';
    state = out_state;
  }
}

I run g++ -std=c++11 -I/home/da.ke/tools/kenlm -L/home/da.ke/tools/kenlm/build/lib -lkenlm_util -lkenlm -lkenlm_builder -lkenlm_filter -lkenlm_interpolate -v -o example example.cpp I've added all .a's just to make sure I didn't miss anything.

The issue is (along with other undefined reference) undefined reference to lm::ngram::Config::Config() I've verified it compiles. Seems like just a linking error, but all those .a's are in place.

Full log as follows: Using built-in specs. COLLECT_GCC=g++ COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.10' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) COLLECT_GCC_OPTIONS='-std=c++11' '-I' '/home/da.ke/tools/kenlm' '-L/home/da.ke/tools/kenlm/build/lib' '-v' '-o' 'example' '-shared-libgcc' '-mtune=generic' '-march=x86-64' /usr/lib/gcc/x86_64-linux-gnu/5/cc1plus -quiet -v -I /home/da.ke/tools/kenlm -imultiarch x86_64-linux-gnu -D_GNU_SOURCE example.cpp -quiet -dumpbase example.cpp -mtune=generic -march=x86-64 -auxbase example -std=c++11 -version -fstack-protector-strong -Wformat -Wformat-security -o /tmp/ccAeGSVF.s GNU C++11 (Ubuntu 5.4.0-6ubuntu1~16.04.10) version 5.4.0 20160609 (x86_64-linux-gnu) compiled by GNU C version 5.4.0 20160609, GMP version 6.1.0, MPFR version 3.1.4, MPC version 1.0.3 GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/5" ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu" ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/include"

include "..." search starts here:

include <...> search starts here:

/home/da.ke/tools/kenlm /usr/include/c++/5 /usr/include/x86_64-linux-gnu/c++/5 /usr/include/c++/5/backward /usr/lib/gcc/x86_64-linux-gnu/5/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed /usr/include/x86_64-linux-gnu /usr/include End of search list. GNU C++11 (Ubuntu 5.4.0-6ubuntu1~16.04.10) version 5.4.0 20160609 (x86_64-linux-gnu) compiled by GNU C version 5.4.0 20160609, GMP version 6.1.0, MPFR version 3.1.4, MPC version 1.0.3 GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 Compiler executable checksum: 61363a0467975df68838dbafa37c997a COLLECT_GCC_OPTIONS='-std=c++11' '-I' '/home/da.ke/tools/kenlm' '-L/home/da.ke/tools/kenlm/build/lib' '-v' '-o' 'example' '-shared-libgcc' '-mtune=generic' '-march=x86-64' as -v -I /home/da.ke/tools/kenlm --64 -o /tmp/ccOJR8Rc.o /tmp/ccAeGSVF.s GNU assembler version 2.26.1 (x86_64-linux-gnu) using BFD version (GNU Binutils for Ubuntu) 2.26.1 COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS='-std=c++11' '-I' '/home/da.ke/tools/kenlm' '-L/home/da.ke/tools/kenlm/build/lib' '-v' '-o' 'example' '-shared-libgcc' '-mtune=generic' '-march=x86-64' /usr/lib/gcc/x86_64-linux-gnu/5/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/5/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper -plugin-opt=-fresolution=/tmp/cccYMnRJ.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o example /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/5/crtbegin.o -L/home/da.ke/tools/kenlm/build/lib -L/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/5/../../.. -lkenlm_util -lkenlm -lkenlm_builder -lkenlm_filter -lkenlm_interpolate /tmp/ccOJR8Rc.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/5/crtend.o /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crtn.o /tmp/ccOJR8Rc.o: In function main': example.cpp:(.text+0x26): undefined reference tolm::ngram::Config::Config()' /tmp/ccOJR8Rc.o: In function util::scoped_memory::reset()': example.cpp:(.text._ZN4util13scoped_memory5resetEv[_ZN4util13scoped_memory5resetEv]+0x23): undefined reference toutil::scoped_memory::reset(void, unsigned long, util::scoped_memory::Alloc)' /tmp/ccOJR8Rc.o: In function lm::ngram::detail::HashForVocab(StringPiece const&)': example.cpp:(.text._ZN2lm5ngram6detail12HashForVocabERK11StringPiece[_ZN2lm5ngram6detail12HashForVocabERK11StringPiece]+0x2f): undefined reference tolm::ngram::detail::HashForVocab(char const, unsigned long)' /tmp/ccOJR8Rc.o: In function lm::ngram::BinaryFormat::~BinaryFormat()': example.cpp:(.text._ZN2lm5ngram12BinaryFormatD2Ev[_ZN2lm5ngram12BinaryFormatD5Ev]+0x48): undefined reference toutil::scoped_fd::~scoped_fd()' /tmp/ccOJR8Rc.o: In function lm::ngram::ProbingModel::ProbingModel(char const*, lm::ngram::Config const&)': example.cpp:(.text._ZN2lm5ngram12ProbingModelC2EPKcRKNS0_6ConfigE[_ZN2lm5ngram12ProbingModelC5EPKcRKNS0_6ConfigE]+0x27): undefined reference tolm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch, lm::ngram::ProbingVocabulary>::GenericModel(char const, lm::ngram::Config const&)' /tmp/ccOJR8Rc.o: In function lm::base::ModelFacade<lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch<lm::ngram::BackoffValue>, lm::ngram::ProbingVocabulary>, lm::ngram::State, lm::ngram::ProbingVocabulary>::~ModelFacade()': example.cpp:(.text._ZN2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_ED2Ev[_ZN2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_ED5Ev]+0x20): undefined reference tolm::base::Model::~Model()' /tmp/ccOJR8Rc.o:(.rodata._ZTIN2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_EE[_ZTIN2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_EE]+0x10): undefined reference to typeinfo for lm::base::Model' /tmp/ccOJR8Rc.o:(.rodata._ZTIN2lm5ngram17ProbingVocabularyE[_ZTIN2lm5ngram17ProbingVocabularyE]+0x10): undefined reference totypeinfo for lm::base::Vocabulary' /tmp/ccOJR8Rc.o: In function lm::ngram::ProbingVocabulary::~ProbingVocabulary()': example.cpp:(.text._ZN2lm5ngram17ProbingVocabularyD2Ev[_ZN2lm5ngram17ProbingVocabularyD5Ev]+0x20): undefined reference tolm::base::Vocabulary::~Vocabulary()' /tmp/ccOJR8Rc.o: In function `lm::base::ModelFacade<lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch, lm::ngram::ProbingVocabulary>, lm::ngram::State, lm::ngram::ProbingVocabulary>::BaseFullScore(void const, unsigned int, void) const': example.cpp:(.text._ZNK2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_E13BaseFullScoreEPKvjPv[_ZNK2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_E13BaseFullScoreEPKvjPv]+0x44): undefined reference to lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch<lm::ngram::BackoffValue>, lm::ngram::ProbingVocabulary>::FullScore(lm::ngram::State const&, unsigned int, lm::ngram::State&) const' /tmp/ccOJR8Rc.o: In functionlm::base::ModelFacade<lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch, lm::ngram::ProbingVocabulary>, lm::ngram::State, lm::ngram::ProbingVocabulary>::BaseFullScoreForgotState(unsigned int const, unsigned int const, unsigned int, void) const': example.cpp:(.text._ZNK2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_E24BaseFullScoreForgotStateEPKjSD_jPv[_ZNK2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_E24BaseFullScoreForgotStateEPKjSD_jPv]+0x50): undefined reference to lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch<lm::ngram::BackoffValue>, lm::ngram::ProbingVocabulary>::FullScoreForgotState(unsigned int const*, unsigned int const*, unsigned int, lm::ngram::State&) const' /tmp/ccOJR8Rc.o: In functionlm::base::ModelFacade<lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch, lm::ngram::ProbingVocabulary>, lm::ngram::State, lm::ngram::ProbingVocabulary>::Score(lm::ngram::State const&, unsigned int, lm::ngram::State&) const': example.cpp:(.text._ZNK2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_E5ScoreERKSAjRSA[_ZNK2lm4base11ModelFacadeINS_5ngram6detail12GenericModelINS3_12HashedSearchINS2_12BackoffValueEEENS2_17ProbingVocabularyEEENS2_5StateES8_E5ScoreERKSAjRSA]+0x40): undefined reference to `lm::ngram::detail::GenericModel<lm::ngram::detail::HashedSearch, lm::ngram::ProbingVocabulary>::FullScore(lm::ngram::State const&, unsigned int, lm::ngram::State&) const' collect2: error: ld returned 1 exit status

Thoughts?

kpu commented 5 years ago

Put the .cpp code before the libraries.

g++ example.cpp -std=c++11 -I/home/da.ke/tools/kenlm -L/home/da.ke/tools/kenlm/build/lib -lkenlm_util -lkenlm -lkenlm_builder -lkenlm_filter -lkenlm_interpolate -v -o example

nicolaskodak commented 5 years ago

@kpu

thanks for prompt reply. Tried your suggestion but no luck; i.e. I run g++ example.cpp -std=c++11 -I/home/da.ke/tools/kenlm -L/home/da.ke/tools/kenlm/build/lib -lkenlm_util -lkenlm -lkenlm_builder -lkenlm_filter -lkenlm_interpolate -v -o example

Every piece of code resides under /home/da.ke/tools/kenlm, where

this time, undefined reference occurs with a bunch of other things, such as undefined reference to util::Pool::~Pool()

By the way, I use the latest approach to build (mkdir build -> cmake .. -> make -j4 ). build_binary and query are tested and they work properly.

Hmm I still have no idea why this is so.

kpu commented 5 years ago

You also have the libraries in reverse order.

nicolaskodak commented 5 years ago

@kpu Right. I wasn't careful about the order. I've used Xlinker to resolve it. Feel free to close this issue. Thanks!

digital10111 commented 5 years ago

Can you please elaborate as to how was this solved. I am facing the same issue. And I am using CMake to use kenlm.