[suggestion] Upgrade to Nuitka instead of PyInstaller

[SirDank]

• Improved performance: Nuitka can significantly improve the performance of Python code by compiling it to a binary format that can be executed more efficiently than interpreted code. This can result in faster execution times and better resource utilization.

[scito]

Thanks for the suggestion. I did not know Nuitka(GitHub). I'll have a look at it. I'm wondering if Nuitka can also include binary dependencies as PyInstaller. I think speed of the extract_otp_secrets application is secondary as it is a "one time usage" application.

[SirDank]

You can make one-time-use applications with nuitka similar to pyinstaller with the --onefile option and you should be able to include any type of file as well.

[scito]

I've tested it locally:

python3.11 -m nuitka --enable-plugin=tk-inter --enable-plugin=pyqt5 --include-data-dir=$HOME/.local/__yolo_v3_qr_detector/=__yolo_v3_qr_detector/ --onefile src/extract_otp_secrets.py

Nuitka is working, however the performance difference to PyInstaller is not very big for time ./extract_otp_secrets.bin -h on my i7-4500U 1.8GHz (Haswell, 4th Gen) computer:

Type	Startup time
PyInstaller	real 0m2.142s, user 0m1.942s, sys 0m0.501s
Nuitka	real 0m1.789s, user 0m1.378s, sys 0m0.708s

[scito]

The build times of pyinstaller and nuitka are very different: Nuitka is about 30x slower to build, 36min instead of 1min.

Measured on my i7-4500U 1.8GHz (Haswell, 4th Gen) computer:

Type	Build time
PyInstaller	real 1m14.388s, user 0m59.825s, sys 0m2.429s
Nuitka	real 35m45.049s, user 25m49.055s, sys 0m59.472s

Currently, it is not feasible to use Nuitka for building.

[SirDank]

That difference is wild, could it be possible you're missing some flags that speed things up, for example: --noinclude-default-mode=nofollow

If you need me to try and make a nuitka build and see how long it takes, just shoot me a message!

[scito]

I rerun it (did not clear the caches):

Type	Build time
PyInstaller	real 0m1.546s, user 0m0.678s, sys 0m0.416s
Nuitka with flag	real 8m0.421s,user 14m28.326s, sys 0m14.239s

Cleaned caches, but not ccache:

Type	Build time
PyInstaller	real real 1m13.567s, user 0m57.239s, sys 0m2.090s
Nuitka with flag	real 6m5.396s, user 7m23.616s, sys 0m7.716s

So, it looks much better.

I get some warnings:

Nuitka-Plugins:WARNING: pyqt5: For the obsolete PyQt5 the Nuitka support is incomplete. Threading, callbacks to compiled
Nuitka-Plugins:WARNING: functions, etc. may not be working.
Nuitka-Plugins:WARNING:     Complex topic! More information can be found at https://nuitka.net/info/pyqt5.html
Nuitka: Starting Python compilation with Nuitka '2.3' on Python '3.11' commercial grade 'not installed'.
Nuitka-Plugins:anti-bloat: Not including '_bisect' automatically in order to avoid bloat, but this may cause: may slow down by
Nuitka-Plugins:anti-bloat: using fallback implementation.
Nuitka-Plugins:anti-bloat: Not including '_json' automatically in order to avoid bloat, but this may cause: may slow down by
Nuitka-Plugins:anti-bloat: using fallback implementation.
Nuitka-Plugins:WARNING: pyqt5: Unwanted import of 'tkinter' that is redundant with 'PyQt5' encountered. Use
Nuitka-Plugins:WARNING: '--nofollow-import-to=tkinter' or uninstall it for best compatibility with pure Python execution.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.                                        
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler: gcc (gcc 7).
Nuitka-Scons: Backend linking program with 257 files (no progress information available for this stage).
Nuitka-Scons: Compiled 256 C files using ccache.
Nuitka-Scons: Cached C files (using ccache) with result 'cache hit': 241
Nuitka-Scons: Cached C files (using ccache) with result 'cache miss': 15
Nuitka-Plugins:dll-files: Found 1 file DLLs from cv2 installation.

[scito]

A try with clang:

Type	Build time
Nuitka cold	real 12m8.076s, user 28m43.976s, sys 0m32.064s
Nuitka warm	real 5m24.256s, user 5m0.535s, sys 0m4.787s

[SirDank]

Here's some more flags you could use: --clang --warn-unusual-code --assume-yes-for-downloads --show-anti-bloat-changes --plugin-enable=pylint-warnings --noinclude-pytest-mode=nofollow --noinclude-setuptools-mode=nofollow --noinclude-default-mode=nofollow

Onefile: --onefile --enable-plugin=upx --upx-binary=path_goes_here --onefile-tempdir-spec={TEMP}\\custom_name_{PID}