search

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.5k stars 513 forks source link

ERROR: Failed building wheel for kenlm (Python 3.9.13 & 3.10.9) #421

Open TrinhNhatTuyen opened 1 year ago

TrinhNhatTuyen commented 1 year ago

Hi, I used !pip install https://github.com/kpu/kenlm/archive/master.zip to install kenlm, and the error is:

Installing build dependencies ... done
 Getting requirements to build wheel ... done
 Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: kenlm
Building wheel for kenlm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for kenlm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [74 lines of output]
      'bash' is not recognized as an internal or external command,
      operable program or batch file.
      'bash' is not recognized as an internal or external command,
      operable program or batch file.
      'bash' is not recognized as an internal or external command,
      operable program or batch file.
      running bdist_wheel
      running build
      running build_ext
      -- Building for: NMake Makefiles
      CMake Error at CMakeLists.txt:14 (project):
        Generator

          NMake Makefiles

        does not support platform specification, but platform

          x64

        was specified.

      CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
      CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
      -- Configuring incomplete, errors occurred!

. . .

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for kenlm
Failed to build kenlm
ERROR: Could not build wheels for kenlm, which is required to install pyproject.toml-based projects

I tried with Python 3.9.13 & 3.10.9 but i didn't work. is there any way to fix it?

kpu commented 1 year ago

What platform is this that doesn't have bash installed?

TheWorldIsNot commented 1 year ago

I get same problem,but have no solution

kpu commented 1 year ago

Install bash...

simozhu commented 1 year ago

I got the same trouble

michael-newsrx commented 1 year ago 
Building wheels for collected packages: sentence_transformers, kenlm, audioread, pylatexenc
  Building wheel for sentence_transformers (setup.py) ... done
  Created wheel for sentence_transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125925 sha256=2a549fa103869746a8ecc40d4fa0c9b3e96a765dbc7a75f559fd7ca00741d490
  Stored in directory: /home/michael/.cache/pip/wheels/ff/27/bf/ffba8b318b02d7f691a57084ee154e26ed24d012b0c7805881
  Building wheel for kenlm (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [89 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building 'kenlm' extension
      creating build
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/lm
      creating build/temp.linux-x86_64-cpython-311/python
      creating build/temp.linux-x86_64-cpython-311/util
      creating build/temp.linux-x86_64-cpython-311/util/double-conversion
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/bhiksha.cc -o build/temp.linux-x86_64-cpython-311/lm/bhiksha.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/binary_format.cc -o build/temp.linux-x86_64-cpython-311/lm/binary_format.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      lm/binary_format.cc: In member function ‘void lm::ngram::BinaryFormat::FinishFile(const lm::ngram::Config&, lm::ngram::ModelType, unsigned int, const std::vector<long unsigned int>&)’:
      lm/binary_format.cc:261:9: warning: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct lm::ngram::Parameters’ with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
        261 |   memset(&params, 0, sizeof(Parameters));
            |   ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from lm/binary_format.cc:1:
      lm/binary_format.hh:42:8: note: ‘struct lm::ngram::Parameters’ declared here
         42 | struct Parameters {
            |        ^~~~~~~~~~
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/config.cc -o build/temp.linux-x86_64-cpython-311/lm/config.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/lm_exception.cc -o build/temp.linux-x86_64-cpython-311/lm/lm_exception.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/model.cc -o build/temp.linux-x86_64-cpython-311/lm/model.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/quantize.cc -o build/temp.linux-x86_64-cpython-311/lm/quantize.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/read_arpa.cc -o build/temp.linux-x86_64-cpython-311/lm/read_arpa.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/search_hashed.cc -o build/temp.linux-x86_64-cpython-311/lm/search_hashed.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/search_trie.cc -o build/temp.linux-x86_64-cpython-311/lm/search_trie.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/sizes.cc -o build/temp.linux-x86_64-cpython-311/lm/sizes.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/trie.cc -o build/temp.linux-x86_64-cpython-311/lm/trie.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/trie_sort.cc -o build/temp.linux-x86_64-cpython-311/lm/trie_sort.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/value_build.cc -o build/temp.linux-x86_64-cpython-311/lm/value_build.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/virtual_interface.cc -o build/temp.linux-x86_64-cpython-311/lm/virtual_interface.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c lm/vocab.cc -o build/temp.linux-x86_64-cpython-311/lm/vocab.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      gcc -pthread -B /home/michael/miniconda3/envs/scholarly-editions/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -O2 -isystem /home/michael/miniconda3/envs/scholarly-editions/include -fPIC -I. -I/home/michael/miniconda3/envs/scholarly-editions/include/python3.11 -c python/kenlm.cpp -o build/temp.linux-x86_64-cpython-311/python/kenlm.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_CLOCKGETTIME
      python/kenlm.cpp: In function ‘void __Pyx_AddTraceback(const char*, int, int, const char*)’:
      python/kenlm.cpp:452:62: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
        452 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      python/kenlm.cpp:8778:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
       8778 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/Python.h:42,
                       from python/kenlm.cpp:6:
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
         22 | typedef struct _frame PyFrameObject;
            |                ^~~~~~
      python/kenlm.cpp: In function ‘PyObject* __Pyx_Coroutine_SendEx(__pyx_CoroutineObject*, PyObject*, int)’:
      python/kenlm.cpp:10248:14: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
      10248 |             f->f_back = PyThreadState_GetFrame(tstate);
            |              ^~
      In file included from /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/Python.h:42,
                       from python/kenlm.cpp:6:
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
         22 | typedef struct _frame PyFrameObject;
            |                ^~~~~~
      In file included from /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/Python.h:38,
                       from python/kenlm.cpp:6:
      python/kenlm.cpp: In function ‘void __Pyx_Coroutine_ResetFrameBackpointer(__Pyx_ExcInfoStruct*)’:
      python/kenlm.cpp:10285:19: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
      10285 |         Py_CLEAR(f->f_back);
            |                   ^~
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/pyport.h:24:38: note: in definition of macro ‘_Py_CAST’
         24 | #define _Py_CAST(type, expr) ((type)(expr))
            |                                      ^~~~
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/object.h:581:29: note: in expansion of macro ‘_PyObject_CAST’
        581 |         PyObject *_py_tmp = _PyObject_CAST(op); \
            |                             ^~~~~~~~~~~~~~
      python/kenlm.cpp:10285:9: note: in expansion of macro ‘Py_CLEAR’
      10285 |         Py_CLEAR(f->f_back);
            |         ^~~~~~~~
      In file included from /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/Python.h:42,
                       from python/kenlm.cpp:6:
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
         22 | typedef struct _frame PyFrameObject;
            |                ^~~~~~
      In file included from /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/Python.h:44,
                       from python/kenlm.cpp:6:
      python/kenlm.cpp:10285:19: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
      10285 |         Py_CLEAR(f->f_back);
            |                   ^~
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/object.h:583:14: note: in definition of macro ‘Py_CLEAR’
        583 |             (op) = NULL;                        \
            |              ^~
      In file included from /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/Python.h:42,
                       from python/kenlm.cpp:6:
      /home/michael/miniconda3/envs/scholarly-editions/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
         22 | typedef struct _frame PyFrameObject;
            |                ^~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for kenlm
  Running setup.py clean for kenlm
  Building wheel for audioread (setup.py) ... done
  Created wheel for audioread: filename=audioread-3.0.0-py3-none-any.whl size=23704 sha256=d76c32a34c53300e5c282bfafc5b9a72439f82c12201c01ea2e0d8899356ba87
  Stored in directory: /home/michael/.cache/pip/wheels/34/6d/02/40eff8045c8287806e7d83333301c1c7bab6109045424e8558
  Building wheel for pylatexenc (setup.py) ... done
  Created wheel for pylatexenc: filename=pylatexenc-2.10-py3-none-any.whl size=136819 sha256=a2e16ff1dfaa941c29808a22f8ac8a9b1c432d867062e5217d22128f60e1e3cf
  Stored in directory: /home/michael/.cache/pip/wheels/b1/7a/33/9fdd892f784ed4afda62b685ae3703adf4c91aa0f524c28f03
Successfully built sentence_transformers audioread pylatexenc
Failed to build kenlm
ERROR: Could not build wheels for kenlm, which is required to install pyproject.toml-based projects
michael-newsrx commented 1 year ago 
echo $SHELL
/bin/bash
kpu commented 1 year ago

@michael-newsrx different issue, looks like cython is broken. Try now that I've updated cython. Or cd kenlm/python && cython -3 --cplus kenlm.pyx yourself.

chrisma2021 commented 1 year ago

I got the same trouble.

Python version information: Python 3.11 (64-bit)

OS version information: OS Name Microsoft Windows 10 Home Version 10.0.19045 Build 19045

PIP version information: Name: pip Version: 23.2.1 Summary: The PyPA recommended tool for installing Python packages. Home-page: https://pip.pypa.io/ Author: The pip developers Author-email: distutils-sig@python.org License: MIT

boots-coder commented 1 year ago

What platform is this that doesn't have bash installed?

Windows10

GNishanth7 commented 1 year ago

I am also getting the same error

kpu commented 1 year ago

Windows is a community-supported platform. I don't support it but you're welcome to fix it.

boots-coder commented 1 year ago

you can install the *whl which is compiled in some webs;i fixxed that on Fasttext by this way

razi-tm commented 11 months ago

https://stackoverflow.com/questions/29846087/error-microsoft-visual-c-14-0-is-required-unable-to-find-vcvarsall-bat

check out the answers provided here. it's for a different question but can solve the problem of kenlm installation on windows.

@TrinhNhatTuyen @TheWorldIsNot @simozhu @chrisma2021 @GNishanth7

quyet12308 commented 11 months ago

https://stackoverflow.com/questions/29846087/error-microsoft-visual-c-14-0-is-required-unable-to-find-vcvarsall-bat

kiểm tra các câu trả lời được cung cấp ở đây. đó là một câu hỏi khác nhưng có thể giải quyết vấn đề cài đặt kenlm trên windows.

@TrinhNhatTuyen @TheWorldIsNot @simozhu @chrisma2021 @GNishanth7

Can you give more details? I tried the methods but I still can't install kenlm for windows

razi-tm commented 11 months ago

download and run visual studio installer and install desktop development with C++, it worked for me.

https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community&channel=Release&version=VS2022&source=VSLandingPage&cid=2030&passive=false

@quyet12308

Lizi-12 commented 8 months ago

download and run visual studio installer and install desktop development with C++, it worked for me.

https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community&channel=Release&version=VS2022&source=VSLandingPage&cid=2030&passive=false

@quyet12308

It works for me! Thanks!

shibing624 commented 6 months ago

use python3.8, and it work for me. 

python3.8 -m pip install kenlm
GoodCabbage commented 5 months ago

install about Microsoft Visual C++ 14.0，it works