Numpy Conda currently slower than Numpy pip on Apple Silicon

[vlebert]

Solution to issue cannot be found in the documentation.

• [X] I checked the documentation.

Issue

I did a few benchmark reported here

https://github.com/numpy/numpy/issues/24961

Numpy install	benchmark script	Results
2. conda install	dario.py	35
3. pip install	dario.py	38
2. conda install	mysvd.py	11.44
3. pip install	mysvd.py	2.71

Installed packages

# packages in environment at /Users/vlebert/test_numpy/conda_install_2023-10-19:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2023.7.22            hf0a4a13_0    conda-forge
libblas                   3.9.0           19_osxarm64_openblas    conda-forge
libcblas                  3.9.0           19_osxarm64_openblas    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_1    conda-forge
libgfortran5              13.2.0               hf226fd6_1    conda-forge
liblapack                 3.9.0           19_osxarm64_openblas    conda-forge
libopenblas               0.3.24          openmp_hd76b1f2_0    conda-forge
libsqlite                 3.43.2               h091b4b1_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
llvm-openmp               17.0.3               hcd81f8e_0    conda-forge
ncurses                   6.4                  h7ea286d_0    conda-forge
numpy                     1.26.0          py312h696b312_0    conda-forge
openssl                   3.1.3                h53f4e23_0    conda-forge
pip                       23.3               pyhd8ed1ab_0    conda-forge
python                    3.12.0          h47c9636_0_cpython    conda-forge
python_abi                3.12                    4_cp312    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.13               hb31c410_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

Environment info

active environment : /Users/vlebert/test_numpy/conda_install_2023-10-19
    active env location : /Users/vlebert/test_numpy/conda_install_2023-10-19
            shell level : 3
       user config file : /Users/vlebert/.condarc
 populated config files : /Users/vlebert/miniforge3/.condarc
                          /Users/vlebert/.condarc
          conda version : 23.9.0
    conda-build version : not installed
         python version : 3.9.13.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.0=0
                          __unix=0=0
       base environment : /Users/vlebert/miniforge3  (writable)
      conda av data dir : /Users/vlebert/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/vlebert/miniforge3/pkgs
                          /Users/vlebert/.conda/pkgs
       envs directories : /Users/vlebert/miniforge3/envs
                          /Users/vlebert/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.9.0 requests/2.31.0 CPython/3.9.13 Darwin/23.0.0 OSX/14.0
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[rgommers]

This needs support for the new Accelerate in macOS >=13.3 - the relevant issue is https://github.com/conda-forge/blas-feedstock/issues/103.

[gsgxnet]

That new Accelerate PR is still open and the checker says there are some conflicts. Will the merge ever happen or is this issue & PR abandoned?

[h-vetinari]

Please be patient, we'll get there.

[gsgxnet]

Sorry, I did not want to put any pressure on anybody. I am very grateful for all the work mostly volunteers put into developing fine open source software. So I am generally patient. May be I worded my question wrong. I just had installed a new Conda-forge environment. And had to learn that the former suggestion to install numpy there on Apple silicon by:
conda install -f numpy "libblas=*=*accelerate" seems to be outdated. So I wanted to clarify whether I should wait for the PR to be merged sometime in the future or better uninstall that numpy and follow the other option.

[h-vetinari]

conda install -f numpy "libblas=*=*accelerate" seems to be outdated.

No, it's currently the best approach, probably 2-4x faster than OpenBLAS (at least, those numbers were cited in #253).

So I wanted to clarify whether I should wait for the PR to be merged sometime in the future or better uninstall that numpy and follow the other option.

Once that PR gets merged, you'll be able to do conda install -f numpy "libblas=*=*newaccelerate". Until then, you can either use the existing *accelerate builds, or install numpy through pip. Note that the latter is not supported by conda-forge, so if something breaks, there's not much we can do.

[gsgxnet]

For fellow users - I had to learn, to keep your libblas setup as intended, you have to make sure that every conda update is done in a way which makes sure that the accelerate setup is kept. E.g.: conda update --all "libblas=*=*accelerate". Otherwise updates might deliver different, unwanted BLAS libs. Confirmation that the updates will run as intended can be verified in the package plan:

...
  added / updated specs:
    - libblas[build=*accelerate]

otherwise these packages would be installed (downgraded):

  libopenblas        conda-forge/osx-arm64::libopenblas-0.3.25-openmp_h6c19121_0 

The following packages will be DOWNGRADED:

  libblas                      3.9.0-20_osxarm64_accelerate --> 3.9.0-20_osxarm64_openblas 
  libcblas                     3.9.0-20_osxarm64_accelerate --> 3.9.0-20_osxarm64_openblas 
  liblapack                    3.9.0-20_osxarm64_accelerate --> 3.9.0-20_osxarm64_openblas 

[dopplershift]

@gsgxnet You can accomplish this by adding the libblas=*=*accelerate in the file (you probably will need to create it) ~/miniconda3/envs/myenv/conda-meta/pinned where miniconda3 is wherever you installed conda and myenv is the environment you want to configure with the pin.

[gsgxnet]

@gsgxnet You can accomplish this by adding the libblas=*=*accelerate in the file (you probably will need to create it) ~/miniconda3/envs/myenv/conda-meta/pinned where miniconda3 is wherever you installed conda and myenv is the environment you want to configure with the pin.

Thanks for that hint. Step by step this thread becomes a useful documentation how to accelerate Apple Silicon Python environments. Great.