Separate and make optional KenLM components

jacobkahn commented 1 year ago

Separates KenLM-related components of Flashlight Text into a separate library: libflashlight-text-kenlm. This library is built when FL_TEXT_USE_KENLM is enabled. In particular, with a shared lib configuration, no KenLM symbols are bundled into libflashlight-text-kenlm.

When building Python bindings, Flashlight Text tries to find kenlm installed via pip (vis-a-vis pip install git@https://github.com/kpu/kenlm). This PR adds machinery to download KenLM headers when building wheels from source but still links to the kenlm installed via pip. This enables dynamically loading a KenLM built from pip by a consuming C++ library from FL Text.

Some other details/changes:

Don't install KenLM headers if not building with it in FL Text.
Adds a new flashlight_lib_text_decoder_kenlm pybind extension.
Moves decoder.py into decoder/__init__.py
The KenLM class in Python bindings now lives in flashlight.lib.text.decoder.kenlm rather than just in decoder.
Since the KenLM installed when building from source with C++/CMake also contains libkenlm_utils (which doesn't exist in the pip installed version), KENLM_DISABLE_UTILS_SEARCH disables searching in Findkenlm for libkenlm_utils since those symbols are all bundled into libkenlm.
If KenLM isn't installed when it should be (or Flashlight Text wheels built without KenLM are used to try to import), things fail with clear telemetry telling the user what's up/how to install KenLM

NB: as above, the import path for KenLM support has changed. It's now:

from flashlight.lib.text.decoder.kenlm import KenLM
# was previously:
from flashlight.lib.text.decoder import KenLM

In order for this to work fully on macOS, https://github.com/kpu/kenlm/pull/413 needs to be merged in because FL Text needs to link at build time with a KenLM dylib and at runtime with the same dylib per portable rpath. Linking with a macOS so bundle at runtime sadly won't work because of how flashlight-text-kenlm was linked.

Test plan: CI, cibuildwheel and local wheel tests across macOS and Linux.

Checklist

[x] Test coverage
[x] Tests pass
[x] Code formatted
[x] Rebased on latest matter
[x] Code documented

mthrok commented 1 year ago

no KenLM symbols are bundled into libflashlight-text-kenlm.

In KenLM.cpp, I see lm::ngram::State and lm::ngram::State are used, so don't they show up as undefined symbols?