search

taan-burapol / nuitka-numba-workaround

Optimizing Python Performance using Numba JIT and Nuitka AOT Compilation
MIT License
0 stars 0 forks source link

JIT compilation not activated during execution of Numpy-importing functions #1

Open alamatti opened 9 months ago

alamatti commented 9 months ago

Hi, thank you for the great work! I am trying to compile a script with Nuitka for standalone distribution, and I have 3 functions heavily relying on Numpy that in the Python execution are accelerated with Numba jit compiler (nopython, of course; I tried with forceobj and it was waaaaay slower). I put the three Numpy-Numba functions in a single separate file in the same folder as the main file, and call it with "exec" as in your example scripts (I actually duplicated the "import" statements in a separate file so I can later obfuscate everything with Marshal). When I compile according to your instructions I obtain execution times similar to vanilla Nuitka-compiled, that is about 6 times slower than Numba jit (3 minutes vs 30 seconds). For completeness and reproducibility, here is the command I used for compilation: python -m nuitka myScript.py --follow-imports --standalone --include-module=numba --module-parameter=numba-disable-jit=no --include-data-file=./numba_import.py=numba_import.py --include-data-file=./numba_jit.py=numba_jit.py

As Nuitka was giving a warning about Numba jit being disabled in Nuitka standalone compilation, I added the flag --module-parameter=numba-disable-jit=no to the previous command, but in this case the compiled exe is not even running, and I receive errors in the function "error_rewrite" in the file "numba\core\dispatcher.py"

I am running on a Windows 10 system, in a conda environment with Python 3.10.13 and the following packages installed via pip (to avoid MKL bloating): numpy==1.26.3 scipy==1.11.4 numba==0.59.0 ordered-set==4.1.0 nuitka==2.0.1 simpleitk==2.3.1 numpy-stl==3.1.1

Have you experienced anything similar? Is it possible to run your proposed hybrid compilation approach when imports are required? Or simply something has changed inside Nuitka after your compilation guide? Thank you in advance.

taan-burapol commented 7 months ago

I am not sure that something has changed since I benchmarked this project in python 3.9, but from your problem in hybrid compilation that you need to obfuscate the import. Maybe you can import in the another function but you need to make sure to call the import function first and then passing the local_dict from that function that you call import to global_dict first before you run the script. Then when you execute bytecode just passing to the script you want.