possible regression? cx_freeze 6.16 - 7.1 for possibly scipy

[jontis]

We have a scientific app based on QT and the scipy stack, compiling with cx_freeze for linux and windows targets. It worked fine with dev version cx_freeze 6.16 that was available during 2023 december. trying to compile now with another system with cx_freeze 7.1, it fails. We have compared the environments and believe that the only difference is versi5on of cx_freeze but can absolutely have missed something. We don't dare to mess with the only functioning build system we have now. The main packages are:

python 3.10
scikit-learn              1.3.2           py310h1fdf081_1    conda-forge
numpy                     1.26.0          py310hb13e2d6_0    conda-forge
pandas                    2.1.3           py310hcc13569_0    conda-forge
pyqt                      5.15.9          py310h04931ad_5    conda-forge
pyqt5-sip                 12.12.2         py310hc6cd4ac_5    conda-forge
qt-main                   5.15.8              h01ceb2d_12    conda-forge

We build with: python setup.py bdist_appimage > cxlog.txt

setup.py:

import toml  # for reading pyproject.toml

pyproject = toml.load("../pyproject.toml")
version = pyproject['project']['version']

print(f"sys.platform: {sys.platform}")
base = "Win32GUI" if sys.platform == "win32" else None

script_path = "main.py"
include_files = ["lib/", ("../pyproject.toml", "lib/pyproject.toml"),] 
exe = Executable(
    script=script_path,
)
# Setup cx_Freeze options.
options = {
    "build_exe": {
        "includes": [],
        "excludes": [],
        "packages": ["pyabf", "igor2", "tqdm", "sklearn", "scipy", "seaborn"],
        "include_files": include_files
    }
}

# Call the setup function.
setup(
    name="brainwash",
    version=version,
    description="",
    #packages=find_packages(where="src"),
    #package_dir={"": "src"},
    #include_package_data=True,
    options=options,
    executables=[exe],
    base=base
)

The error when trying to run appimage built by cx_freeze 7.1 is: $ dist/brainwash-0.9.0-x86_64.AppImage /tmp/.mount_brainwHL6FMz/main: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

[marcelotduarte]

Please test with cx_Freeze 7.1.1 I have tested some samples of pandas, scikit-learn, using a newer version, and they have worked. For instance, I have tested, using scikit-learn 1.5.0 and an example from sklearn auto_examples_python/bicluster, used bdist_appimage and it works.

[jontis]

Tested with cx_freeze 7.1.1 from conda. Same problem. It says very early in the process: sys.platform: linux running bdist_appimage /home/jonathan/.local/bin/appimagetool --version running build_exe patchelf --version returns: 'patchelf 0.17.2\n' WARNING: cannot find 'libcrypt.so.2'

I've checked and while libcrypt.so.2 is present in several other env, it is not in this env. Missing dependancy for packaging?

[jontis]

I solved that error by: mamba install libxcrypt this provided libcrypt.so.2

Now on to next error. File "/home/jonathan/mambaforge/envs/brainwash/lib/python3.10/site-packages/scipy/linalg/blas.py", line 213, in from scipy.linalg import _fblas ImportError: libcblas.so.3: cannot open shared object file: No such file or directory

This file is already present in env: $ locate libcblas.so.3 /home/jonathan/mambaforge/envs/brainwash/lib/libcblas.so.3

And the only mention of it in the build log is: $ cat cxlog.txt | grep libcblas patchelf --replace-needed libcblas.so.3 libopenblasp-r0.3.24.so /home/jonathan/code/brainwash/src/build/exe.linux-x86_64-3.10/lib/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so returns: ''

[marcelotduarte]

Try to upgrade numpy cx_Freeze have a sample: https://github.com/marcelotduarte/cx_Freeze/tree/main/samples/pandas I tested it with numpy 1.26.4 and pandas 2.2.2 and works. You can use it to test your environment.

[jontis]

Tried but no change. It may be scipy that is the problem, and it seems to work for cx_freeze 6.16 Error message: File "/home/jonathan/mambaforge/envs/brainwash/lib/python3.10/site-packages/scipy/linalg/blas.py", line 213, in from scipy.linalg import _fblas ImportError: libcblas.so.3: cannot open shared object file: No such file or directory

Triggered by: from scipy.signal import savgol_filter, find_peaks

[jontis]

Tried also bumping scipy to latest 1.13.1 but the error remains unchanged.

[jontis]

Checked for differences in blas in both functioning cxfreeze 6.16 and non functioning cxfreeze 7.1.1 system but they are the same: libblas 3.9.0 19_linux64_openblas conda-forge libcblas 3.9.0 19_linux64_openblas conda-forge liblapack 3.9.0 19_linux64_openblas conda-forge libopenblas 0.3.24 pthreads_h413a1c8_0 conda-forge

[marcelotduarte]

Few questions: 1) The pandas sample works in your side? 2) How you have installed numpy? 3) Can you test using: 3.1) conda install "libblas==mkl" numpy or 3.2) conda install "libblas==openblas" numpy 4) You can indicate a sample/example of code using scipy to test on my side?

[marcelotduarte]

Tried also bumping scipy to latest 1.13.1 but the error remains unchanged.

In your first comment of this thread, you do not listed the scipy version. Checking the sources, the lasted tested version is 1.11.2. Can you test with it? I'll try find a example to test the latest version.

[jontis]

Ok, results of the tests:

4. to test, I just added to the top of your pandas sample test_pandas.py: from scipy import signal 1, your pandas sample works both in its minimal env and our production env.
5. linux comes with openblas as default, and we're running that for both linux and windows as the mkl is huge with very little performance gains. 3.1 you line doesn't work to switch to mkl, but this did: conda install "libblas==openblas" numpy. The pandas sample with scipy signal builds but does not run. It crashes with: Traceback (most recent call last): File "/home/jonathan/mambaforge/envs/cxtest/lib/python3.12/site-packages/cx_Freeze/initscripts/startup.py", line 141, in run module_init.run(name + "main") File "/home/jonathan/mambaforge/envs/cxtest/lib/python3.12/site-packages/cx_Freeze/initscripts/console.py", line 25, in run exec(code, main_globals) File "test_pandas.py", line 5, in from scipy import signal ... ... from scipy.linalg import _fblas ImportError: libcblas.so.3: cannot open shared object file: No such file or directory 3.2 builds with openblas but crashes the same way as did with mkl

[marcelotduarte]

3.1 for me is not working too w/ the pandas sample. In the docs for scipy signal: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.correlate.html#scipy.signal.correlate I get a example to test scipy 1.11.2 and scipy 1.13.1 fails I'll work on this.

[jontis]

We tried to compile this example with the cxfreeze 6.16 dev and it seems to work fine.