Fail to find KenLM when installing the python binding

[Achronferry]

Question

I try to install the Python bindings by python setup.py install. However, it fails to find the KenLM even if it has already been installed.

Additional Context

The error log is as following, How should I set the environment variable like CMAKE_LIBRARY_PATH, KENLM_LIB or KENLM_ROOT……

python setup.py install
running install
/DB/rhome/chenyuyang/miniconda3/envs/cuda113/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/DB/rhome/chenyuyang/miniconda3/envs/cuda113/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing flashlight_text.egg-info/PKG-INFO
writing dependency_links to flashlight_text.egg-info/dependency_links.txt
writing top-level names to flashlight_text.egg-info/top_level.txt
package init file 'flashlight/__init__.py' not found (or not a regular file)
package init file 'flashlight/lib/text/__init__.py' not found (or not a regular file)
reading manifest file 'flashlight_text.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'flashlight_text.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- Looking for KenLM
-- kenlm library not found; if you already have kenlm installed, please set CMAKE_LIBRARY_PATH, KENLM_LIB or KENLM_ROOT environment variable
-- kenlm utils library not found; if you already have kenlm installed, please set CMAKE_LIBRARY_PATH, KENLM_UTIL_LIB or KENLM_ROOT environment variable
-- kenlm model.hh not found; if you already have kenlm installed, please set CMAKE_INCLUDE_PATH, KENLM_MODEL_HEADER or KENLM_ROOT environment variable
-- Could NOT find kenlm (missing: KENLM_LIBRARIES) 
CMake Error at flashlight/lib/text/decoder/lm/CMakeLists.txt:21 (message):
  KenLM not found but FL_TEXT_USE_KENLM enabled.  Install KenLM or set the
  KENLM_ROOT environment variable.
Call Stack (most recent call first):
  flashlight/lib/text/decoder/CMakeLists.txt:3 (include)
  flashlight/lib/text/CMakeLists.txt:8 (include)
  CMakeLists.txt:54 (include)

I've added export KENLM_ROOT=/DB/rhome/chenyuyang/tools/kenlm/build/bin but it seems not work..

[Achronferry]

I changed the KENLM_ROOT to /DB/rhome/chenyuyang/tools/kenlm/build/ and it could find KenLM, while another error comes:

/GPFS/data/chenyuyang/projects/flashlight/flashlight/lib/text/decoder/lm/KenLM.cpp:12:10: fatal error: kenlm/lm/model.hh: No such file or directory

[jacobkahn]

@Achronferry — the easiest way to resolve this is to make install KenLM — you should build it with a custom CMAKE_INSTALL_PREFIX then install, and use that base directory as your KENLM_ROOT.

If you can't install for whatever reason, can you try using export KENLM_ROOT= /DB/rhome/chenyuyang/tools/kenlm? It should find the built libs in the build directory.

[Achronferry]

@Achronferry — the easiest way to resolve this is to make install KenLM — you should build it with a custom CMAKE_INSTALL_PREFIX then install, and use that base directory as your KENLM_ROOT.

If you can't install for whatever reason, can you try using export KENLM_ROOT= /DB/rhome/chenyuyang/tools/kenlm? It should find the built libs in the build directory.

Thanks for your reply! I guess it fail to find model.hh just because I didn't set the CMAKE_INSTALL_PREFIX . I changes the #include <kenlm/lm/model.hh> in flashlight/flashlight/lib/text/decoder/lm/KenLM.cpp to the absolute path and it also works.

[jacobkahn]

@Achronferry — glad it's working. This is an artifact of how the source are laid out in KenLM (that they aren't contained in a kenlm directory) so the header paths aren't always internally consistent post-install depending on the prefix. Let me know if you continue having issues; I'm hoping to make some improvements to KenLM to fix this issue.