search

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.46k stars 514 forks source link

ERROR: Could not build wheels for kenlm, which is required to install pyproject.toml-based projects #438

Open solee0022 opened 11 months ago

solee0022 commented 11 months ago

I want to install kenlm, but I got this error. And I've already installed 'pyproject.toml'. How can I fix this issue?

(asr2) $ pip install https://github.com/kpu/kenlm/archive/master.zip
Collecting https://github.com/kpu/kenlm/archive/master.zip
  Using cached https://github.com/kpu/kenlm/archive/master.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: kenlm
  Building wheel for kenlm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for kenlm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [218 lines of output]
      Will build with KenLM max_order set to 6
      running bdist_wheel
      running build
      running build_ext
      CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.

      -- The C compiler identification is NVHPC 22.2.0
      -- The CXX compiler identification is NVHPC 22.2.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/bin/nvc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/bin/nvc++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Could NOT find Eigen3 (missing: Eigen3_DIR)
      CMake Deprecation Warning at python/BuildStandalone.cmake:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.
      Call Stack (most recent call first):
        CMakeLists.txt:31 (include)

      -- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
      -- Found BZip2: /usr/lib64/libbz2.so (found version "1.0.6")
      -- Looking for BZ2_bzCompressInit
      -- Looking for BZ2_bzCompressInit - found
      -- Looking for lzma_auto_decoder in /usr/lib64/liblzma.so
      -- Looking for lzma_auto_decoder in /usr/lib64/liblzma.so - found
      -- Looking for lzma_easy_encoder in /usr/lib64/liblzma.so
      -- Looking for lzma_easy_encoder in /usr/lib64/liblzma.so - found
      -- Looking for lzma_lzma_preset in /usr/lib64/liblzma.so
      -- Looking for lzma_lzma_preset in /usr/lib64/liblzma.so - found
      -- Found LibLZMA: /usr/lib64/liblzma.so (found version "5.2.4")
      -- Configuring done (1.7s)
      -- Generating done (0.0s)
      -- Build files have been written to: /tmp/pip-req-build-8tk25yr5/build/temp.linux-x86_64-cpython-39
      [  2%] Building CXX object CMakeFiles/kenlm.dir/lm/bhiksha.cc.o
      [  4%] Building CXX object CMakeFiles/kenlm.dir/lm/binary_format.cc.o
      [  9%] Building CXX object CMakeFiles/kenlm.dir/lm/lm_exception.cc.o
      [  9%] Building CXX object CMakeFiles/kenlm.dir/lm/config.cc.o
      [ 12%] Building CXX object CMakeFiles/kenlm.dir/lm/model.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/state.hh", line 77: warning: pointer points outside of underlying object
            for (uint64_t * i = pointers + length; i < pointers + KENLM_MAX_ORDER - 1; ++i)
                                                                ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      [ 14%] Building CXX object CMakeFiles/kenlm.dir/lm/quantize.cc.o
      [ 17%] Building CXX object CMakeFiles/kenlm.dir/lm/read_arpa.cc.o
      [ 19%] Building CXX object CMakeFiles/kenlm.dir/lm/search_hashed.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/state.hh", line 77: warning: pointer points outside of underlying object
            for (uint64_t * i = pointers + length; i < pointers + KENLM_MAX_ORDER - 1; ++i)
                                                                ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      [ 21%] Building CXX object CMakeFiles/kenlm.dir/lm/search_trie.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/search_trie.cc", line 180: warning: pointer points outside of underlying object
              for (BackoffMessages *i = messages_; i != messages_ + KENLM_MAX_ORDER - 1; ++i)
                                                                  ^

      "/tmp/pip-req-build-8tk25yr5/lm/search_trie.cc", line 430: warning: variable "backoffs" was declared but never referenced
          std::vector<float> probs, backoffs;
                                    ^
                detected during:
                  instantiation of "void lm::ngram::trie::BuildTrie(lm::ngram::trie::SortedFiles &, std::vector<uint64_t, std::allocator<uint64_t>> &, const lm::ngram::Config &, lm::ngram::trie::TrieSearch<Quant, Bhiksha> &, Quant &, lm::ngram::SortedVocabulary &, lm::ngram::BinaryFormat &) [with Quant=lm::ngram::DontQuantize, Bhiksha=lm::ngram::trie::DontBhiksha]" at line 587
                  instantiation of "void lm::ngram::trie::TrieSearch<Quant, Bhiksha>::InitializeFromARPA(const char *, util::FilePiece &, std::vector<uint64_t, std::allocator<uint64_t>> &, const lm::ngram::Config &, lm::ngram::SortedVocabulary &, lm::ngram::BinaryFormat &) [with Quant=lm::ngram::DontQuantize, Bhiksha=lm::ngram::trie::DontBhiksha]" at line 590

      [ 24%] Building CXX object CMakeFiles/kenlm.dir/lm/sizes.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/state.hh", line 77: warning: pointer points outside of underlying object
            for (uint64_t * i = pointers + length; i < pointers + KENLM_MAX_ORDER - 1; ++i)
                                                                ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      [ 26%] Building CXX object CMakeFiles/kenlm.dir/lm/trie.cc.o
      [ 29%] Building CXX object CMakeFiles/kenlm.dir/lm/trie_sort.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/trie_sort.cc", line 124: warning: parameter "entry_size" was declared but never referenced
          void operator()(std::size_t entry_size, unsigned char order, const void *first, const void *second, FILE * /*out*/) const {
                                      ^

      "/tmp/pip-req-build-8tk25yr5/lm/trie_sort.cc", line 124: warning: parameter "second" was declared but never referenced
          void operator()(std::size_t entry_size, unsigned char order, const void *first, const void *second, FILE * /*out*/) const {
                                                                                                      ^

      "/tmp/pip-req-build-8tk25yr5/lm/trie_sort.cc", line 241: warning: variable "deleter" was declared but never referenced
                util::scoped_FILE deleter(*i);
                                  ^

      "/tmp/pip-req-build-8tk25yr5/lm/trie_sort.cc", line 246: warning: variable "deleter" was declared but never referenced
              util::scoped_FILE deleter(files_.front());
                                ^

      [ 31%] Building CXX object CMakeFiles/kenlm.dir/lm/value_build.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/state.hh", line 77: warning: pointer points outside of underlying object
            for (uint64_t * i = pointers + length; i < pointers + KENLM_MAX_ORDER - 1; ++i)
                                                                ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      [ 34%] Building CXX object CMakeFiles/kenlm.dir/lm/virtual_interface.cc.o
      [ 36%] Building CXX object CMakeFiles/kenlm.dir/lm/vocab.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 67: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::SortedVocabulary"
        class SortedVocabulary : public base::Vocabulary {
              ^

      "/tmp/pip-req-build-8tk25yr5/lm/vocab.hh", line 156: warning: overloaded virtual function "lm::base::Vocabulary::Index" is only partially overridden in class "lm::ngram::ProbingVocabulary"
        class ProbingVocabulary : public base::Vocabulary {
              ^

      [ 39%] Building CXX object CMakeFiles/kenlm.dir/python/score_sentence.cc.o
      "/tmp/pip-req-build-8tk25yr5/lm/state.hh", line 77: warning: pointer points outside of underlying object
            for (uint64_t * i = pointers + length; i < pointers + KENLM_MAX_ORDER - 1; ++i)
                                                                ^

      [ 41%] Building CXX object CMakeFiles/kenlm.dir/util/bit_packing.cc.o
      [ 43%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/bignum-dtoa.cc.o
      [ 46%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/bignum.cc.o
      [ 48%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/cached-powers.cc.o
      [ 51%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/double-to-string.cc.o
      [ 53%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/fast-dtoa.cc.o
      [ 56%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/fixed-dtoa.cc.o
      [ 58%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/string-to-double.cc.o
      [ 60%] Building CXX object CMakeFiles/kenlm.dir/util/double-conversion/strtod.cc.o
      "/tmp/pip-req-build-8tk25yr5/util/double-conversion/strtod.cc", line 562: warning: variable "f2" was set but never used
          float f2 = float_guess;
                ^

      [ 63%] Building CXX object CMakeFiles/kenlm.dir/util/ersatz_progress.cc.o
      "/tmp/pip-req-build-8tk25yr5/util/double-conversion/strtod.cc", line 457: warning: function "double_conversion::AssertTrimmedDigits" was declared but never referenced
        static bool AssertTrimmedDigits(const Vector<const char>& buffer) {
                    ^

      [ 65%] Building CXX object CMakeFiles/kenlm.dir/util/exception.cc.o
      [ 68%] Building CXX object CMakeFiles/kenlm.dir/util/file.cc.o
      [ 70%] Building CXX object CMakeFiles/kenlm.dir/util/file_piece.cc.o
      [ 73%] Building CXX object CMakeFiles/kenlm.dir/util/float_to_string.cc.o
      [ 75%] Building CXX object CMakeFiles/kenlm.dir/util/integer_to_string.cc.o
      [ 78%] Building CXX object CMakeFiles/kenlm.dir/util/mmap.cc.o
      [ 80%] Building CXX object CMakeFiles/kenlm.dir/util/murmur_hash.cc.o
      "/tmp/pip-req-build-8tk25yr5/util/murmur_hash.cc", line 163: warning: function "util::<unnamed>::MurmurHashNativeBackend<L>(const void *, std::size_t, uint64_t) [with L=4U]" was declared but never referenced
        template <> inline uint64_t MurmurHashNativeBackend<4>(const void * key, std::size_t len, uint64_t seed) {
                                    ^

      [ 82%] Building CXX object CMakeFiles/kenlm.dir/util/parallel_read.cc.o
      [ 85%] Building CXX object CMakeFiles/kenlm.dir/util/pool.cc.o
      [ 87%] Building CXX object CMakeFiles/kenlm.dir/util/read_compressed.cc.o
      "/tmp/pip-req-build-8tk25yr5/util/mmap.cc", line 323: warning: statement is unreachable
              break;
              ^

      [ 90%] Building CXX object CMakeFiles/kenlm.dir/util/scoped.cc.o
      [ 92%] Building CXX object CMakeFiles/kenlm.dir/util/spaces.cc.o
      [ 95%] Building CXX object CMakeFiles/kenlm.dir/util/string_piece.cc.o
      [ 97%] Building CXX object CMakeFiles/kenlm.dir/util/usage.cc.o
      [100%] Linking CXX shared library /tmp/pip-req-build-8tk25yr5/build/lib.linux-x86_64-cpython-39/libkenlm.so
      [100%] Built target kenlm
      building 'kenlm' extension
      creating build/temp.linux-x86_64-cpython-39/lm
      creating build/temp.linux-x86_64-cpython-39/python
      creating build/temp.linux-x86_64-cpython-39/util
      creating build/temp.linux-x86_64-cpython-39/util/double-conversion
      /opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/bin/nvc -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/solee0022/anaconda3/envs/asr2/include -I/home/solee0022/anaconda3/envs/asr2/include -fPIC -O2 -isystem /home/solee0022/anaconda3/envs/asr2/include -fPIC -I. -I/home/solee0022/anaconda3/envs/asr2/include/python3.9 -c lm/bhiksha.cc -o build/temp.linux-x86_64-cpython-39/lm/bhiksha.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
      nvc-Error-Unknown switch: -Wno-unused-result
      error: command '/opt/ohpc/pub/apps/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/bin/nvc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for kenlm
Failed to build kenlm
ERROR: Could not build wheels for kenlm, which is required to install pyproject.toml-based projects
lapolonio commented 9 months ago

looks related https://kheafield.com/code/kenlm/dependencies/

tnq177 commented 3 months ago

Installing boost & eigen fixed this for me. Inside a conda env:

conda install conda-forge::boost
conda install conda-forge::eigen
pip install https://github.com/kpu/kenlm/archive/master.zip