search

python-control / Slycot

Python wrapper for the Subroutine Library in Systems and Control Theory (SLICOT)
Other
130 stars 42 forks source link

Installing Slycot (openmp) breaks numpy #230

Open chnce opened 7 months ago

chnce commented 7 months ago

After installing slycot in a conda environment, a kernel crash occurs when trying to solve a linear system with a singular matrix.

I narrowed the issue down to the following:

Openmp places its "libiomp5md.dll" in Library\bin. The version from numpy is either in Library\bin (channel main) or directly in bin (conda-forge).

Actual Behavior

Using numpy.linalg.solve on singular matrix with dimension over 149 crashes the Kernel.

Expected Behavior

Expected a "LinAlgError: Singular matrix" as it is shown with matrices up to the dimension of 149.

Steps to Reproduce

import numpy as np

n = 150
A = np.random.rand(n,n)
A[-1,:] = np.zeros((1,n)) # make matrix singular

b = np.random.rand(n)

x = np.linalg.solve(A, b)
moorepants commented 7 months ago

How are you installing the various packages? All with conda? Or other ways?

chnce commented 7 months ago

Thanks for the quick response. I detailed the steps in the original post above.

moorepants commented 7 months ago

Given that you are reporting a clash with numpy and openmp binaries from conda forge, then I think you should open the issue at one or both of these locations:

https://github.com/conda-forge/openmp-feedstock

https://github.com/conda-forge/numpy-feedstock

I don't see a slycot specific issue (yet).

chnce commented 7 months ago

I agree that it's not a true issue of slycot. But maybe slycot could move to a more modern version of libflang. I have little hope, that anyone at openmp is going to fix a 6 year old version.

I could frankenstein together a working environment by ignoring the dependencies and installing:

moorepants commented 7 months ago

Ah, ok I understand now. slycot forces an old libflang and thus forces you to install an old openmp, which has an incompatibility with numpy. So yes, it would be helpful to move slycot to a newer libflang.

It doesn't look like this is something we explicitly set:

https://github.com/conda-forge/slycot-feedstock/blob/main/recipe/meta.yaml

It may mean that conda forge requires libfang<6 from the {{ compiler('fortran') }} call. If so, then this would be a bug to report to the numpy and openmp feedstocks.

moorepants commented 7 months ago

It looks locked to libflang 5 on windows across all of conda-forge: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml#L30

moorepants commented 7 months ago

Related: https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/5081

bnavigator commented 7 months ago

Yes we try to follow the conda-forge pinning for flang. See also https://github.com/conda-forge/slycot-feedstock/pull/65 and https://github.com/conda-forge/blas-feedstock/issues/106

chnce commented 7 months ago

Thank you for your insights. I filed the issue with numpy and openmp feedstock. I'm genuinely surprised that no one else seems to have the same issue. openmp 5 was downloaded half a million times from conda-forge.

robertobucher commented 6 months ago

There is a similar problem under Debian. Using the installed python3-numpy package (numpy version 1.24.2 in trixie)

pip install slycot

doesn't work correctly (building stops). The only way to install slycot ist to force the removing of python3-numpy

sudo dpkg -P --force-all python3-numpy)

and install numpy using pip (in this case the installed version is numpy-1.26.4).After this step

pip install spycot

works.

bnavigator commented 6 months ago

Could you post a build log with the error message, please?

robertobucher commented 6 months ago

Tested on a docker after clonng Slycot in git, but it correclty works after updating numpy. In the pyproject.toml the version of rerquired numpy is numpy >= 1.23.1 an in Debian we have 1.24.2.

root@debian:/psc/Slycot# pip install --break-system-packages . Processing /psc/Slycot Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: numpy>=1.23.1 in /usr/lib/python3/dist-packages (from slycot==0.5.5.dev60+gfea37c7) (1.24.2) Building wheels for collected packages: slycot Building wheel for slycot (pyproject.toml) ... error error: subprocess-exited-with-error

�� Building wheel for slycot (pyproject.toml) did not run successfully. ��� exit code: 1 ������> [166 lines of output]

  --------------------------------------------------------------------------------
  -- Trying 'Ninja' generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 3.5 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.

  Not searching for unused variables given on the command line.

  CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  -- Configuring incomplete, errors occurred!
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying 'Ninja' generator - failure
  --------------------------------------------------------------------------------

  --------------------------------------------------------------------------------
  -- Trying 'Unix Makefiles' generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 3.5 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.

  Not searching for unused variables given on the command line.

  -- The C compiler identification is GNU 13.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The Fortran compiler identification is GNU 13.2.0
  -- Detecting Fortran compiler ABI info
  -- Detecting Fortran compiler ABI info - done
  -- Check for working Fortran compiler: /usr/bin/gfortran - skipped
  -- Configuring done (0.2s)
  -- Generating done (0.0s)
  -- Build files have been written to: /psc/Slycot/_cmake_test_compile/build
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying 'Unix Makefiles' generator - success
  --------------------------------------------------------------------------------

  Configuring Project
    Working directory:
      /psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-build
    Command:
      /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/cmake/data/bin/cmake /psc/Slycot -G 'Unix Makefiles' --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.8 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.11.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DCMAKE_BUILD_TYPE:STRING=Release

  Not searching for unused variables given on the command line.
  -- The C compiler identification is GNU 13.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The Fortran compiler identification is GNU 13.2.0
  -- Detecting Fortran compiler ABI info
  -- Detecting Fortran compiler ABI info - done
  -- Check for working Fortran compiler: /usr/bin/gfortran - skipped
  -- Found Python: /usr/bin/python3 (found version "3.11.8") found components: Interpreter Development NumPy Development.Module Development.Embed
  CMake Warning (dev) at /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:245 (find_package):
    Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
    are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
    the cmake_policy command to set the policy and suppress this warning.

  Call Stack (most recent call first):
    CMakeLists.txt:13 (find_package)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Found PythonInterp: /usr/bin/python3 (found version "3.11.8")
  CMake Warning (dev) at /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:252 (find_package):
    Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
    are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
    the cmake_policy command to set the policy and suppress this warning.

  Call Stack (most recent call first):
    CMakeLists.txt:13 (find_package)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.11.so (found version "3.11.8")
  -- Found NumPy: /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include (found version "1.26.4")
  Traceback (most recent call last):
    File "/usr/bin/f2py3", line 33, in <module>
      sys.exit(load_entry_point('numpy==1.24.2', 'console_scripts', 'f2py3.11')())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/bin/f2py3", line 25, in importlib_load_entry_point
      return next(matches).load()
             ^^^^^^^^^^^^^
  StopIteration
  -- Found F2PY: /usr/bin/f2py3 (found version "")
  -- Looking for Fortran sgemm
  -- Looking for Fortran sgemm - not found
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE
  -- Looking for Fortran sgemm
  -- Looking for Fortran sgemm - found
  -- Found BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so
  -- Looking for Fortran cheev
  -- Looking for Fortran cheev - found
  -- Found LAPACK: /usr/lib/x86_64-linux-gnu/libopenblas.so;-lm;-ldl
  -- Python headers included from: /usr/include/python3.11
  -- NumPy headers included from: /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include
  -- F2PY headers included from: /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/f2py/src;/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include
  -- LAPACK: /usr/lib/x86_64-linux-gnu/libopenblas.so;-lm;-ldl
  -- BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so
  CMake Error at CMakeLists.txt:29 (message):
    Python include directory has a numpy sub-directory,

          /usr/include/python3.11/numpy,
      which is different from Numpy include directory
        /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include.
      You're probably building in a virtual environment, in which case
      uninstall numpy from the base environment and try again.

  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/setuptools_wrap.py", line 666, in setup
      env = cmkr.configure(
            ^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/cmaker.py", line 357, in configure
      raise SKBuildError(msg)

  An error occurred while configuring with CMake.
    Command:
      /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/cmake/data/bin/cmake /psc/Slycot -G 'Unix Makefiles' --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.8 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.11.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DCMAKE_BUILD_TYPE:STRING=Release
    Source directory:
      /psc/Slycot
    Working directory:
      /psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-build
  Please see CMake's output for more information.

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for slycot Failed to build slycot ERROR: Could not build wheels for slycot, which is required to install pyproject.toml-based projects root@debian:/psc/Slycot#