search

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.51k stars 511 forks source link

KenLM Import Error Python3.9 #379

Open patrickvonplaten opened 2 years ago

patrickvonplaten commented 2 years ago

When installing KenLM on the following environment:

Linux pop-os 5.15.15-76051515-generic #202201160435~1642693824~21.10~97db1bb SMP Thu Jan 20 17:35:05 U x86_64 x86_64 x86_64 GNU/Linux

and Python version:

Python 3.9.7

I cannot import kenLM afterwards as it errors out. E.g. the following does not work:

pip install https://github.com/kpu/kenlm/archive/master.zip

and 

import kenLM

gives:

ImportError: /home/patrick/anaconda3/envs/hf/lib/python3.9/site-packages/kenlm.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv

Could it be related to the recent changes in CPython: https://github.com/kpu/kenlm/commit/0760f4c4df76f3286656e7232dc3ad6495248bc2 ?

patrickvonplaten commented 2 years ago

It does work though with Python 3.8

patrickvonplaten commented 2 years ago

Also gently pinging @xantares in case this looks familiar :sweat_smile:

patrickvonplaten commented 2 years ago

Any updates here?

xantares commented 2 years ago

it works for me on my side with 3.9 when I rebuild it from sources

patrickvonplaten commented 2 years ago

Sorry to ask here again. By rebuilding from sources you mean that you don't use pip install https://github.com/kpu/kenlm/archive/master.zip ?

xantares commented 2 years ago

yes; calling cmake directly

kpu commented 2 years ago

The pip version does a poor man's compilation globbing the cc files instead of cmake. Is this a subtle C++ version mismatch?

kpu commented 2 years ago

Ugh seems like a more fun problem with gcc and python https://github.com/stan-dev/pystan/issues/294

patrickvonplaten commented 2 years ago

Uff yeah I see thanks a lot for the pointers!

patrickvonplaten commented 2 years ago

Conda was the problem for me it seems. Even from source the installation didn't work.

The solution as shown here: https://github.com/stan-dev/pystan/issues/294#issuecomment-878292636 was the right fix though for me :-) Now it works! Thanks @kpu for the link!