numpy is not binary compatible with libflang/openmp 5

chnce commented 10 months ago

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue

The original issue was filed here: https://github.com/python-control/Slycot/issues/230

After installing libflang 5 in a conda environment, a kernel crash occurs when trying to solve a linear system with a singular matrix.

Flang 5 is pinned as the default Fortran compiler across conda-forge. conda_build_config.yaml

I narrowed the issue down to the following:

libflang 5.0.0 depends on openmp 5.0.0
openmp 5.0.0 overwrites "libiomp5md.dll" that was installed by numpy. The two dlls are very different in size (2mb for the numpy one, 600kb for the openmp one)

Openmp places its "libiomp5md.dll" in Library\bin. The version from numpy is either in Library\bin (channel main) or directly in bin (conda-forge).

Actual Behavior

Using numpy.linalg.solve on singular matrix with dimension over 149 crashes the Kernel.

Expected Behavior

Expected a "LinAlgError: Singular matrix" as it is shown with matrices up to the dimension of 149.

Steps to Reproduce

Create a new conda environment with python 3.11, numpy (and ipython for convenience)
- mamba create -n slycot_crash python=3.11 numpy ipython -c conda-forge
activate the environment
- mamba activate slycot_crash
enter ipython
- ipython
paste and run the code below to confirm it works as expected (raises LinAlgError)
exit ipython
- exit
install openmp=5
- mamba install openmp=5 -c conda-forge
enter ipython and run the code below again to confirm kernel crash
(optional) change n=149 and it only fails if the size is 150 or larger

import numpy as np

n = 150
A = np.random.rand(n,n)
A[-1,:] = np.zeros((1,n)) # make matrix singular

b = np.random.rand(n)

x = np.linalg.solve(A, b)

Installed packages

# Name                    Version                   Build  Channel
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
bzip2                     1.0.8                hcfcfb64_5    conda-forge
ca-certificates           2023.11.17           h56e8100_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
intel-openmp              2024.0.0         h57928b3_49841    conda-forge
ipython                   8.20.0             pyh7428d3b_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
libblas                   3.9.0              21_win64_mkl    conda-forge
libcblas                  3.9.0              21_win64_mkl    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libhwloc                  2.9.3           default_haede6df_1009    conda-forge
libiconv                  1.17                 hcfcfb64_2    conda-forge
liblapack                 3.9.0              21_win64_mkl    conda-forge
libsqlite                 3.44.2               hcfcfb64_0    conda-forge
libxml2                   2.12.4               hc3477c8_1    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
llvm-meta                 5.0.0                         0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mkl                       2024.0.0         h66d3029_49657    conda-forge
numpy                     1.26.3          py311h0b4df5a_0    conda-forge
openmp                    5.0.0                    vc14_1    conda-forge
openssl                   3.2.0                hcfcfb64_1    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
python                    3.11.7          h2628c8c_1_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tbb                       2021.11.0            h91493d7_1    conda-forge
tk                        8.6.13               h5226925_1    conda-forge
traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                hcf57466_18    conda-forge
vc14_runtime              14.38.33130         h82b7239_18    conda-forge
vs2015_runtime            14.38.33130         hcb4865c_18    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge

Environment info

active environment : slycot_crash
    active env location : C:\Users\\AppData\Local\Continuum\Miniconda3\envs\slycot_crash
            shell level : 3
       user config file : C:\Users\\.condarc
 populated config files : C:\Users\\.condarc
          conda version : 23.3.1
    conda-build version : 3.25.0
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=11.0=0
                          __win=0=0
       base environment : C:\Users\\AppData\Local\Continuum\Miniconda3  (writable)
      conda av data dir : C:\Users\\AppData\Local\Continuum\Miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\\AppData\Local\Continuum\Miniconda3\pkgs
                          C:\Users\\.conda\pkgs
                          C:\Users\\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\\AppData\Local\Continuum\Miniconda3\envs
                          C:\Users\\.conda\envs
                          C:\Users\\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.3.1 requests/2.29.0 CPython/3.11.4 Windows/10 Windows/10.0.22621
          administrator : False
             netrc file : None
           offline mode : False

h-vetinari commented 10 months ago

Why are you installing openmp=5 at all? This has been replaced by llvm-openmp for a long time. The slycot issue also didn't provide a reason why v5.

The unfortunate reality is that some windows bits in conda-forge still rely on llvm-openmp 5, but that'll hopefully change soon, because llvm/flang 18 should now be ready to take over the fortran side.

chnce commented 10 months ago

Installing openmp=5 directly is only a shortcut to reproduce the issue. Originally I only installed slycot. Slycot has libflang 5 as a dependency which then has openmp 5 as dependency.

slycot

{
  "arch": "x86_64",
  "build": "py311h3fb5059_4",
  "build_number": 4,
  "depends": [
    "libblas >=3.9.0,<4.0a0",
    "libcblas >=3.9.0,<4.0a0",
    "libflang >=5.0.0,<6.0.0.a0",
    "liblapack >=3.9.0,<4.0a0",
    "numpy >=1.23.5,<2.0a0",
    "python >=3.11,<3.12.0a0",
    "python_abi 3.11.* *_cp311",
    "ucrt >=10.0.20348.0",
    "vc >=14.2,<15",
    "vc14_runtime >=14.29.30139"
  ],
  "license": "GPL-2.0-only",
  "license_family": "GPL",
  "name": "slycot",
  "platform": "win",
  "subdir": "win-64",
  "timestamp": 1701634917193,
  "version": "0.5.4"
}

libflang

{
  "arch": "x86_64",
  "build": "h6538335_20180525",
  "build_number": 20180525,
  "depends": [
    "openmp 5.0.0",
    "vc >=14,<15.0a0"
  ],
  "license": "Apache 2.0",
  "name": "libflang",
  "platform": "win",
  "subdir": "win-64",
  "timestamp": 1527899216421,
  "track_features": "flang",
  "version": "5.0.0"
}

h-vetinari commented 10 months ago

Yeah, the libflang 5 issue is for stuff on windows compiled with our ancient flang. As soon as we've got the shiny new llvm-flang 18 in conda-forge, I'm planning to give https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/1359 another shot.

jcapriot commented 2 months ago

should the numpy recipe then have a run_constrained blocking the openmp package from getting installed together?

conda-forge / numpy-feedstock