Snips doesn't work with pyinstaller

[Shotgun167]

My team just spent the past week to get snips to work with pyinstaller. Pyinstaller lets you wrap your python code up into a convenient single file for distribution. The modifications to Snips were minor:

snips_nlu_parsers/utils.py: add the import for sys, and change the assignment of PACKAGE_PATH to the following

try:
    PACKAGE_PATH = Path(sys._MEIPASS)
except AttributeError:
    PACKAGE_PATH = Path(__file__).absolute().parent

_MEIPASS is a variable set by pyinstaller that records where the installation is in the environment it creates. The call to load the library a few lines down will break without this.

Add the hook-snips_nlu_parsers.py file:

from PyInstaller.compat import modname_tkinter, is_win
import os

hiddenimports = ['sklearn', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree._utils', 'snips_nlu', 'multiprocessing.get_context', 'sklearn.utils', 'pycrfsuite._dumpparser', 'pycrfsuite._logparser']

if is_win:
    binaries=[(os.environ['USERPROFILE'] + '\\AppData\\Local\\Programs\\python\\python36\\lib\\site-packages\\snips_nlu_parsers\\dylib\\libsnips_nlu_parsers_rs.cp36-win_amd64.pyd', 'dylib')]
else:
    binaries=[('/usr/local/lib/python3.7/site-packages/snips_nlu_parsers/dylib/libsnips_nlu_parsers_rs.cpython-37m-darwin.so', 'dylib')]

Documentation for pyinstaller hooks: https://pyinstaller.readthedocs.io/en/stable/hooks.html?highlight=collect_submodules

[adrienball]

@Shotgun167 , did it work with the changes you described?

[Shotgun167]

One other change that I left out. Within python, when loading the engine, you have to use the same pattern used in utils.py. The code for loading my engine is:

        try:
            PACKAGE_PATH = Path(sys._MEIPASS)
        except AttributeError:
            PACKAGE_PATH = Path(".")
        self.snips_engine = SnipsNLUEngine.from_path(str(PACKAGE_PATH) + "/snips/nebula_engine")

We have built and are using executables for both Windows and Mac, and our Jenkins build machine is running the CI chain on Linux.

[Shotgun167]

Thank you for accepting the change; however. . .

snips_nlu_parsers/utils.py needs "import sys"

I apologize. It's my fault for leaving it out. My team mate made the change on his system and everything works correctly.

[adrienball]

@Shotgun167 Thanks for providing these details, that will be valuable for other users relying on PyInstaller. For the moment, we will leave these fixes out of snips-nlu as it is hard to test and the responsibility seems to be rather on the user side.

[pratyushud]

This answer is very helpful, thank you

[padmalcom]

I still get an error when building with PyInstaller. This seems to be related to the file libsnips_nlu_parsers_rs.cp36-win_amd64.pyd. But I double checked the path the pyd file is loaded from and it is correct. Any ideas?

Traceback (most recent call last):
  File "main_12.py", line 315, in <module>
  File "main_12.py", line 166, in __init__
  File "intentmgmt.py", line 120, in __init__
  File "snips_nlu\common\log_utils.py", line 30, in wrapped
  File "snips_nlu\nlu_engine\nlu_engine.py", line 95, in fit
  File "snips_nlu\dataset\validation.py", line 74, in validate_and_format_dataset
  File "snips_nlu\dataset\validation.py", line 171, in _validate_and_format_custom_entity
  File "snips_nlu\dataset\validation.py", line 105, in _has_any_capitalization
  File "snips_nlu\preprocessing.py", line 96, in tokenize_light
  File "snips_nlu_utils\token.py", line 21, in tokenize_light
  File "ctypes\__init__.py", line 394, in __getattr__
  File "ctypes\__init__.py", line 399, in __getitem__
AttributeError: function 'snips_nlu_utils_tokenize_light' not found

[Shotgun167]

Pyinstaller takes all of your python files and packs them into a zip file.  When you run the program, it unpacks the file to a temp directory, and runs it from there.  The issue you're running into is that it is trying to dynamically load a library.  I've attached several files that are needed for building Snips with Pyinstaller.  The build scripts will copy the update python files over the one's provided by Snips.  Do a diff to see what changed.  Then the hooks files insure that the libraries are included by Pyinstaller.  Our app is called Nebula and it requires the selenium chromedriver.  You can ignore anything dealing with either of those.

On Wednesday, January 6, 2021, 7:57:49 AM EST, padmalcom <notifications@github.com> wrote:  

I still get an error when building with PyInstaller. This seems to be related to the file libsnips_nlu_parsers_rs.cp36-win_amd64.pyd. But I double checked the path the pyd file is loaded from and it is correct. Any ideas? Traceback (most recent call last): File "main_12.py", line 315, in File "main_12.py", line 166, in init File "intentmgmt.py", line 120, in init File "snips_nlu\common\log_utils.py", line 30, in wrapped File "snips_nlu\nlu_engine\nlu_engine.py", line 95, in fit File "snips_nlu\dataset\validation.py", line 74, in validate_and_format_dataset File "snips_nlu\dataset\validation.py", line 171, in _validate_and_format_custom_entity File "snips_nlu\dataset\validation.py", line 105, in _has_any_capitalization File "snips_nlu\preprocessing.py", line 96, in tokenize_light File "snips_nlu_utils\token.py", line 21, in tokenize_light File "ctypes__init.py", line 394, in getattr File "ctypes\init.py", line 399, in getitem__ AttributeError: function 'snips_nlu_utils_tokenize_light' not found

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[padmalcom]

Hi @Shotgun167 thanks for your comment. I'm fixing/adding one missing file after another so that snips_nlu_utils_tokenize_light is found now. Currently, I'm stuck at compiling snips-nlu-ontology on win10 x64 (see https://github.com/snipsco/snips-nlu-ontology/issues/158). I'm wondering why these module's binaries are reqired since I don't find snips-nlu-ontology in my conda environment before packaging the application. Have you had to deal with the ontology module, too?

[Shotgun167]

No.  I've not had a problem with tokenize_light, either.  I haven't even seen that file.  Not sure what is happening with you there.  What version of snips?  v.21 has been working for us, so we've actively avoided any new releases that may have happened.

On Friday, January 8, 2021, 3:04:22 AM EST, padmalcom <notifications@github.com> wrote:  

Hi @Shotgun167 thanks for your comment. I'm fixing/adding one missing file after another so that snips_nlu_utils_tokenize_light is found now. Currently, I'm stuck at compiling snips-nlu-ontology on win10 x64 (see snipsco/snips-nlu-ontology#158). I'm wondering why these module's binaries are reqired since I don't find snips-nlu-ontology in my conda environment before packaging the application. Have you had to deal with the ontology module, too?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[padmalcom]

Hi that's strange, I tried the last two versions and saw that there weren't any updates for snips nlu for quite a long time, so I guess there won't be any support by the developers. Thus, I will stick with anaconda+git instead of using pyinstaller.