Can't run the compiled spacy application: No module named 'spacy.lexeme'

[ovo-Tim]

I'm trying to build a Python script utilizing the spaCy library. The build process completes successfully; however, when I run the compiled application, I encounter an error: No module named 'spacy.lexeme'.

Version Info

Using docker. created by docker run -t -i --network host -v /tmp/testnuitka/:/my_file python /bin/bash

root@TimTu-Arch:/my_file# python -m nuitka --version
2.5rc4
Commercial: None
Python: 3.12.4 (main, Aug  2 2024, 14:41:30) [GCC 12.2.0]
Flavor: Unknown
Executable: /usr/local/bin/python
OS: Linux
Arch: x86_64
Distribution: Debian 12
Version C compiler: /usr/bin/gcc (gcc 12).

Install ways

I've tried the stable version that installed by pip install nuitka, and the dev version that installed by pip install -U "https://github.com/Nuitka/Nuitka/archive/develop.zip". The dev version can't even build:

  File "/usr/local/lib/python3.12/site-packages/nuitka/nodes/NodeBases.py", line 106, in makeClone
    result = self.__class__(source_ref=self.source_ref, **self.getCloneArgs())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: ExpressionBuiltinMakeException.__init__() missing 1 required positional argument: 'for_raise'

The specific PyPI names and versions

(shortened)

root@TimTu-Arch:/my_file# pip list -v
Package            Version  Location                                Installer
------------------ -------- --------------------------------------- ---------
spacy              3.7.5    /usr/local/lib/python3.12/site-packages pip
spacy-legacy       3.0.12   /usr/local/lib/python3.12/site-packages pip
spacy-loggers      1.0.5    /usr/local/lib/python3.12/site-packages pip

Example

The python file:

import spacy

The commands I use:

docker run -t -i --network host -v /tmp/testnuitka/:/my_file python /bin/bash
root@TimTu-Arch:/my_file# ls
test.py
root@TimTu-Arch:/my_file# cat test.py
import spacy
root@TimTu-Arch:/my_file# pip install spacy nuitka
root@TimTu-Arch:/my_file# apt update
root@TimTu-Arch:/my_file# apt install clang
root@TimTu-Arch:/my_file# apt install patchelf
root@TimTu-Arch:/my_file# nuitka --standalone --output-dir=build --clang --script-name=test.py
Nuitka-Options: Used command line options: --standalone --output-dir=build --clang --script-name=test.py
Nuitka: Starting Python compilation with Nuitka '2.4.5' on Python '3.12' commercial grade 'not installed'.
Nuitka-Plugins:WARNING: spacy: No language models included. Use the option '--spacy-language-model=language_model_name' to include one. Use 'all' to include all
Nuitka-Plugins:WARNING: downloaded ones, or select from the list of installed ones:
Nuitka-Plugins:anti-bloat: Not including '_bisect' automatically in order to avoid bloat, but this may cause: may slow down by using fallback implementation.
Nuitka-Plugins:anti-bloat: Not including '_json' automatically in order to avoid bloat, but this may cause: may slow down by using fallback implementation.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler: clang (clang 14.0.6).
Nuitka-Scons: Backend linking program with 1122 files (no progress information available for this stage).
Nuitka-Scons:WARNING: You are not using ccache, re-compilation of identical code will be slower than necessary. Use your OS package manager to install it.
Nuitka-Plugins:data-files: Included data file 'certifi/cacert.pem' due to package data for 'certifi'.
Nuitka-Plugins:data-files: Included data file 'jaraco/text/Lorem ipsum.txt' due to package data for 'jaraco.text'.
Nuitka-Plugins:data-files: Included 52 data files due to package data directory '.' for 'spacy'.
Nuitka-Plugins:data-files: Included data file 'thinc/backends/_custom_kernels.cu' due to package data for 'thinc.backends'.
Nuitka-Plugins:data-files: Included data file 'thinc/backends/_murmur3.cu' due to package data for 'thinc.backends'.
Nuitka: Keeping build directory 'build/test.build'.
Nuitka: Successfully created 'build/test.dist/test.bin'.
root@TimTu-Arch:/my_file# build/test.dist/test.bin
Traceback (most recent call last):
  File "/my_file/build/test.dist/test.py", line 1, in <module>
  File "/my_file/build/test.dist/spacy/__init__.py", line 13, in <module spacy>
  File "/my_file/build/test.dist/spacy/pipeline/__init__.py", line 1, in <module spacy.pipeline>
  File "/my_file/build/test.dist/spacy/pipeline/attributeruler.py", line 8, in <module spacy.pipeline.attributeruler>
  File "/my_file/build/test.dist/spacy/language.py", line 43, in <module spacy.language>
  File "/my_file/build/test.dist/spacy/pipe_analysis.py", line 6, in <module spacy.pipe_analysis>
  File "/my_file/build/test.dist/spacy/tokens/__init__.py", line 1, in <module spacy.tokens>
  File "/my_file/build/test.dist/spacy/tokens/_serialize.py", line 14, in <module spacy.tokens._serialize>
  File "spacy/vocab.pyx", line 1, in init spacy.vocab
ModuleNotFoundError: No module named 'spacy.lexeme'

Build command

nuitka --standalone --output-dir=build --clang --script-name=test.py

[ovo-Tim]

I do remember it could be compiled on older nuitka, I am still testing.

Just failed on 2.3.11:

Nuitka-Plugins:dll-files: Found 2 files DLLs from thinc installation.
FATAL: Error, call to 'patchelf' failed: ['/usr/bin/patchelf', '--set-rpath', '$ORIGIN/../..:$ORIGIN/..', 'build/test.dist/thinc/backends/_custom_kernels.cu'] -> b'patchelf: not an ELF executable'.

[ovo-Tim]

I have tried to add flag --follow-import-to="spacy.lexeme", but that still couldn't solve the issues.

[kayhayen]

That only does follow to it it if asked, for which --standalone already gives permission. It doesn't make its use being noticed, for that --include-module would force that.

[ovo-Tim]

It works now. Thanks #3059.

[kayhayen]

There was a regression for develop, from the 3.12 code genenration optimization that has since been fixed, factory ought to work now and the next hotfix as well. We keep issues open until they are released in a stable release.