conda-forge / bmad-feedstock

A conda-smithy repository for bmad.
BSD 3-Clause "New" or "Revised" License
1 stars 6 forks source link

Latest versions not functional on MacOS due to lapack95 undefined symbol `_clagge_` #461

Closed ken-lauer closed 3 weeks ago

ken-lauer commented 3 weeks ago

Solution to issue cannot be found in the documentation.

Issue

The symbol in question comes from lapack95: https://github.com/conda-forge/lapack95-feedstock/issues/7

pytao and any other libraries which use libtao.dylib will fail with the following message when libblas=*=accelerate is installed:

$ python -c 'import pytao; pytao.Tao(init_file="$ACC_ROOT_DIR/bmad-doc/tao_examples/cbeta_cell/tao.init")'

OSError: dlopen($CONDA/envs/bmad/bin/../lib/libtao.dylib, 0x0006): symbol not found in flat namespace '_clagge_'

Our previous workaround was to use libblas instead (conda install "libblas=*=*_openblas"), however it appears that this downgrades bmad. It's no longer installable with openblas:

$ conda create -n bmad bmad=20240922 libblas=*=*openblas
Channels:
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package libblas-3.9.0-1_openblas requires libopenblas >=0.3.12,<0.3.13.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ bmad 20240922**  is installable and it requires
│  └─ libopenblas >=0.3.28,<1.0a0 , which can be installed;
└─ libblas * *openblas is not installable because there are no viable options
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.17,<0.3.18.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.18,<0.3.19.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.20,<0.3.21.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.21,<0.3.22.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.12,<0.3.13.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.15,<0.3.16.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.23,<0.3.24.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.24,<0.3.25.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.25,<0.3.26.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.26,<0.3.27.0a0 , which conflicts with any installable versions previously reported;
   └─ libblas 3.9.0 would require
      └─ libopenblas >=0.3.27,<0.3.28.0a0 , which conflicts with any installable versions previously reported.

Installed packages

# Name                    Version                   Build  Channel
annotated-types           0.7.0              pyhd8ed1ab_0    conda-forge
bmad                      20240922.0      nompi_hc9d7628_100    conda-forge
brotli                    1.1.0                hd74edd7_2    conda-forge
brotli-bin                1.1.0                hd74edd7_2    conda-forge
bzip2                     1.0.8                h99b78c6_7    conda-forge
c-ares                    1.33.1               hd74edd7_0    conda-forge
ca-certificates           2024.8.30            hf0a4a13_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.18.0               hb4a6bf7_3    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
contourpy                 1.3.0           py312h6142ec9_1    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
expat                     2.6.3                hf9b8971_0    conda-forge
fftw                      3.3.10          nompi_h6637ab6_110    conda-forge
fgsl                      1.6.0                hf716404_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
fontconfig                2.14.2               h82840c6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.53.1          py312h024a12e_1    conda-forge
freetype                  2.12.1               hadb7bae_2    conda-forge
fribidi                   1.0.10               h27ca646_0    conda-forge
graphite2                 1.3.13            hebf3989_1003    conda-forge
gsl                       2.7                  h6e638da_0    conda-forge
h5py                      3.11.0          nompi_py312h903599c_102    conda-forge
harfbuzz                  9.0.0                h997cde5_1    conda-forge
hdf5                      1.14.3          nompi_hec07895_105    conda-forge
icu                       75.1                 hfee45f7_0    conda-forge
kiwisolver                1.4.7           py312h6142ec9_0    conda-forge
krb5                      1.21.3               h237132a_0    conda-forge
lapack95                  3.0.0                hf0628b0_3    conda-forge
lcms2                     2.16                 ha0e7c42_0    conda-forge
lerc                      4.0.0                h9a09cb3_0    conda-forge
libaec                    1.1.3                hebf3989_0    conda-forge
libblas                   3.9.0           24_osxarm64_accelerate    conda-forge
libbrotlicommon           1.1.0                hd74edd7_2    conda-forge
libbrotlidec              1.1.0                hd74edd7_2    conda-forge
libbrotlienc              1.1.0                hd74edd7_2    conda-forge
libcblas                  3.9.0           24_osxarm64_accelerate    conda-forge
libcurl                   8.10.1               h13a7ad3_0    conda-forge
libcxx                    19.1.0               ha82da77_0    conda-forge
libdeflate                1.21                 h99b78c6_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h93a5062_2    conda-forge
libexpat                  2.6.3                hf9b8971_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_3    conda-forge
libgfortran5              13.2.0               hf226fd6_3    conda-forge
libglib                   2.80.3               h59d46d9_2    conda-forge
libiconv                  1.17                 h0d3ecfb_2    conda-forge
libintl                   0.22.5               h8414b35_3    conda-forge
libjpeg-turbo             3.0.0                hb547adb_1    conda-forge
liblapack                 3.9.0           24_osxarm64_accelerate    conda-forge
libnghttp2                1.58.0               ha4dd798_1    conda-forge
libopenblas               0.3.28          openmp_h517c56d_0    conda-forge
libpng                    1.6.44               hc14010f_0    conda-forge
libsqlite                 3.46.1               hc14010f_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libtiff                   4.7.0                h9c1d414_0    conda-forge
libwebp-base              1.4.0                h93a5062_0    conda-forge
libxcb                    1.16                 hc9fafa5_1    conda-forge
libzlib                   1.3.1                hfb2fe0b_1    conda-forge
llvm-openmp               18.1.8               hde57baf_1    conda-forge
matplotlib-base           3.9.2           py312h9bd0bc6_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.5                  h7bae524_1    conda-forge
numpy                     2.1.1           py312h801f5e3_0    conda-forge
openjpeg                  2.5.2                h9f1df11_0    conda-forge
openssl                   3.3.2                h8359307_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pango                     1.54.0               h9ee27a3_2    conda-forge
pcre2                     10.44                h297a79d_2    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pgplot                    5.2.2             hde848ab_1009    conda-forge
pillow                    10.4.0          py312h8609ca0_1    conda-forge
pixman                    0.43.4               hebf3989_0    conda-forge
pthread-stubs             0.4               hd74edd7_1002    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pydantic                  2.9.2              pyhd8ed1ab_0    conda-forge
pydantic-core             2.23.4          py312he431725_0    conda-forge
pyparsing                 3.1.4              pyhd8ed1ab_0    conda-forge
pytao                     0.4.0              pyhd8ed1ab_0    conda-forge
python                    3.12.6          h739c21a_1_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    5_cp312    conda-forge
qhull                     2020.2               h420ef59_5    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h8827d51_1    conda-forge
xorg-kbproto              1.0.7             hd74edd7_1003    conda-forge
xorg-libice               1.1.1                hb547adb_0    conda-forge
xorg-libsm                1.2.4                hb547adb_0    conda-forge
xorg-libx11               1.8.9                he5f3e76_1    conda-forge
xorg-libxau               1.0.11               hd74edd7_1    conda-forge
xorg-libxdmcp             1.1.3                hd74edd7_1    conda-forge
xorg-libxt                1.3.0                hb547adb_1    conda-forge
xorg-xextproto            7.3.0             hd74edd7_1004    conda-forge
xorg-xproto               7.0.31            hd74edd7_1008    conda-forge
xraylib                   4.1.5           py312hbf5b3ca_1    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
zlib                      1.3.1                hfb2fe0b_1    conda-forge
zstd                      1.5.6                hb46c0d2_0    conda-forge

Environment info

conda version : 24.7.1
    conda-build version : 24.5.1
         python version : 3.10.14.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m2
                          __conda=24.7.1=0
                          __osx=14.5=0
                          __unix=0=0
               platform : osx-arm64
             user-agent : conda/24.7.1 requests/2.32.3 CPython/3.10.14 Darwin/23.5.0 OSX/14.5 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
                UID:GID : 501:20
             netrc file : None
           offline mode : False
ken-lauer commented 3 weeks ago

With @hhslepicka we released a new lapack95 that specifies openblas as the variant for MacOS (PR link).

No luck still:

$ conda create -n bmad-test bmad=20240922 libblas=*=*openblas
Channels:
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package libblas-3.9.0-1_openblas requires libopenblas >=0.3.12,<0.3.13.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ bmad 20240922**  is installable and it requires
│  └─ libopenblas >=0.3.28,<1.0a0 , which can be installed;
└─ libblas * *openblas is not installable because there are no viable options
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.17,<0.3.18.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.18,<0.3.19.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.20,<0.3.21.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.21,<0.3.22.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.12,<0.3.13.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.15,<0.3.16.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.23,<0.3.24.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.24,<0.3.25.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.25,<0.3.26.0a0 , which conflicts with any installable versions previously reported;
   ├─ libblas 3.9.0 would require
   │  └─ libopenblas >=0.3.26,<0.3.27.0a0 , which conflicts with any installable versions previously reported;
   └─ libblas 3.9.0 would require
      └─ libopenblas >=0.3.27,<0.3.28.0a0 , which conflicts with any installable versions previously reported.

If I understand this correctly, it's saying bmad wants libopenblas 0.3.28 (which is available on conda-forge), however the latest version of libblas 3.9.0 is only compatible up to 0.3.27.

Is it worth having multiple bmad builds with libopenblas versions? Should we run_constrain libopenblas to <=0.3.27? Or even... is this something that should be reported to libblas/libopenblas? Edit: OK, this is pending upstream in blas-feedstock (https://github.com/conda-forge/blas-feedstock/pull/121)

I'm not sure what the right approach here is.

ken-lauer commented 3 weeks ago

I also bumped this in blas-feedstock as the other linked PR never ran its CI and is a bit behind: https://github.com/conda-forge/blas-feedstock/pull/124

In my (non-expert) opinion, it seems like openblas shouldn't get a release unless blas-feedstock gets an update around the same time. Somehow those got out of lock-step and have caused this issue.

Maybe:

ken-lauer commented 3 weeks ago

It looks like after following the knowledge base suggestion in #463, it's building variants for openblas 0.3.27 and 0.3.28:

2024-09-24T17:45:39.0563833Z     openblas:                  0.3.27-pthreads_h9eca1d5_1     conda-forge
...
2024-09-24T17:19:38.8900447Z     openblas:                  0.3.28-pthreads_hbcdf1e8_0 conda-forge

This was not the case as of the latest main branch build (which only did 0.3.28). So I think this may have been the cause. I'll tweak/simplify that PR a bit more and hopefully this issue will be closed today...

ken-lauer commented 3 weeks ago

It appears to me with the changes in this feedstock (#463) and lapack95, it tends to select openblas and create a valid environment on MacOS arm64. I'll leave this open for a bit in case others still have issues.

ken-lauer commented 3 weeks ago

I think we're good now