setup on M1 Mac

[kelle]

I'm trying to get setup on a new ARM mac and I've hit this problem when trying to create the new environment using the yml file. I've followed this document so far. I'm looping in @tepickering since he helped me get this point.

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package freetype conflicts for:
matplotlib[version='>=3.2'] -> matplotlib-base[version='>=3.3.3,<3.3.4.0a0'] -> freetype[version='>=2.10.4,<3.0a0']
scikit-image[version='>=0.17'] -> matplotlib-base[version='>=2.0.0'] -> freetype[version='>=2.10.4,<3.0a0']

Package pip conflicts for:
pandas[version='>=1.0'] -> python[version='>=3.8,<3.9.0a0'] -> pip
jupyter -> python -> pip
python=3.8 -> pip
cython -> python[version='>=3.9,<3.10.0a0'] -> pip
jupyter-offlinenotebook[version='>=0.1.0'] -> python -> pip
scipy[version='>=1.5'] -> python[version='>=3.8,<3.9.0a0'] -> pip
notebook[version='>=6.0'] -> python[version='>=3.8,<3.9.0a0'] -> pip
scikit-image[version='>=0.17'] -> python[version='>=3.8,<3.9.0a0'] -> pip
numpy[version='>=1.16'] -> python[version='>=3.8,<3.9.0a0'] -> pip
pip
ipython[version='>=7.17'] -> python[version='>=3.8,<3.9.0a0'] -> pip

Package numpy conflicts for:
scikit-image[version='>=0.17'] -> imageio[version='>=2.3.0'] -> numpy[version='>=1.15.1|>=1.19.2,<2.0a0']
scikit-image[version='>=0.17'] -> numpy[version='>=1.19.4,<2.0a0']
numpy[version='>=1.16']
pandas[version='>=1.0'] -> numpy[version='>=1.19.2,<2.0a0|>=1.19.4,<2.0a0']
scipy[version='>=1.5'] -> numpy[version='>=1.19.2,<2.0a0|>=1.19.4,<2.0a0']

Package decorator conflicts for:
scikit-image[version='>=0.17'] -> networkx[version='>=2.0'] -> decorator[version='>=3.4.0|>=4.3.0']
ipython[version='>=7.17'] -> decorator

Package liblapack conflicts for:
pandas[version='>=1.0'] -> numpy[version='>=1.19.4,<2.0a0'] -> liblapack[version='>=3.9.0,<4.0a0']
scipy[version='>=1.5'] -> liblapack[version='>=3.9.0,<4.0a0']
numpy[version='>=1.16'] -> liblapack[version='>=3.9.0,<4.0a0']
scikit-image[version='>=0.17'] -> numpy[version='>=1.19.4,<2.0a0'] -> liblapack[version='>=3.9.0,<4.0a0']

Package setuptools conflicts for:
python=3.8 -> pip -> setuptools
matplotlib[version='>=3.2'] -> matplotlib-base[version='>=3.3.3,<3.3.4.0a0'] -> setuptools
ipython[version='>=7.17'] -> setuptools[version='>=18.5']
scikit-image[version='>=0.17'] -> matplotlib-base[version='>=2.0.0'] -> setuptools
ipython[version='>=7.17'] -> pygments -> setuptools
pip -> setuptools
notebook[version='>=6.0'] -> jinja2 -> setuptools

Package jupyterlab_pygments conflicts for:
notebook[version='>=6.0'] -> nbconvert -> jupyterlab_pygments
jupyter -> nbconvert -> jupyterlab_pygments

Package python_abi conflicts for:
numpy[version='>=1.16'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
pip -> setuptools -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
jupyter-offlinenotebook[version='>=0.1.0'] -> notebook -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
cython -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
scipy[version='>=1.5'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
pandas[version='>=1.0'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
scikit-image[version='>=0.17'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
ipython[version='>=7.17'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
notebook[version='>=6.0'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
jupyter -> ipykernel -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']
matplotlib[version='>=3.2'] -> python_abi[version='3.8.*|3.9.*',build='*_cp39|*_cp38']

Package tornado conflicts for:
scikit-image[version='>=0.17'] -> matplotlib-base[version='>=2.0.0'] -> tornado
notebook[version='>=6.0'] -> ipykernel -> tornado[version='>=4|>=4.0|>=4.2|>=4.1']
jupyter-offlinenotebook[version='>=0.1.0'] -> notebook -> tornado[version='>=5.0']
jupyter -> ipykernel -> tornado[version='>=4.0|>=4.2|>=5.0']
notebook[version='>=6.0'] -> tornado[version='>=5.0']
matplotlib[version='>=3.2'] -> matplotlib-base[version='>=3.3.3,<3.3.4.0a0'] -> tornado

Package libopenblas conflicts for:
scipy[version='>=1.5'] -> libblas[version='>=3.9.0,<4.0a0'] -> libopenblas[version='>=0.3.12,<0.3.13.0a0|>=0.3.12,<1.0a0']
numpy[version='>=1.16'] -> libblas[version='>=3.9.0,<4.0a0'] -> libopenblas[version='>=0.3.12,<0.3.13.0a0|>=0.3.12,<1.0a0']

Package python-dateutil conflicts for:
pandas[version='>=1.0'] -> python-dateutil[version='>=2.7.3']
scikit-image[version='>=0.17'] -> matplotlib-base[version='>=2.0.0'] -> python-dateutil[version='>=2.1']
notebook[version='>=6.0'] -> jupyter_client[version='>=5.3.4'] -> python-dateutil[version='>=2.1']
matplotlib[version='>=3.2'] -> matplotlib-base[version='>=3.3.3,<3.3.4.0a0'] -> python-dateutil[version='>=2.1']

Package six conflicts for:
pandas[version='>=1.0'] -> python-dateutil[version='>=2.7.3'] -> six
ipython[version='>=7.17'] -> prompt-toolkit[version='!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0'] -> six[version='>=1.9.0']
notebook[version='>=6.0'] -> argon2-cffi -> six

Package ptyprocess conflicts for:
ipython[version='>=7.17'] -> pexpect[version='>4.3'] -> ptyprocess[version='>=0.5']
notebook[version='>=6.0'] -> terminado[version='>=0.8.3'] -> ptyprocess

Package packaging conflicts for:
pip -> wheel -> packaging[version='>=20.2']
scikit-image[version='>=0.17'] -> pooch[version='>=1.3.0'] -> packaging

Package ipython conflicts for:
notebook[version='>=6.0'] -> ipykernel -> ipython[version='>=5.0']
jupyter -> ipykernel -> ipython[version='>=4.0.0|>=5.0']
ipython[version='>=7.17']

Package libgfortran5 conflicts for:
numpy[version='>=1.16'] -> libblas[version='>=3.9.0,<4.0a0'] -> libgfortran5[version='>=11.0.0.dev0']
scipy[version='>=1.5'] -> libgfortran5[version='>=11.0.0.dev0']
scikit-image[version='>=0.17'] -> scipy[version='>=0.19'] -> libgfortran5[version='>=11.0.0.dev0']
scipy[version='>=1.5'] -> libgfortran=5 -> libgfortran5

Package notebook conflicts for:
jupyter-offlinenotebook[version='>=0.1.0'] -> notebook
notebook[version='>=6.0']
jupyter -> notebook

Package ipython_genutils conflicts for:
notebook[version='>=6.0'] -> ipython_genutils
jupyter -> notebook -> ipython_genutils
ipython[version='>=7.17'] -> traitlets[version='>=4.2'] -> ipython_genutils
jupyter-offlinenotebook[version='>=0.1.0'] -> notebook -> ipython_genutils

Package scipy conflicts for:
scipy[version='>=1.5']
scikit-image[version='>=0.17'] -> scipy[version='>=0.19']

Package certifi conflicts for:
pip -> setuptools -> certifi[version='>=2016.9.26']
matplotlib[version='>=3.2'] -> matplotlib-base[version='>=3.3.3,<3.3.4.0a0'] -> certifi[version='>=2020.06.20']
ipython[version='>=7.17'] -> setuptools[version='>=18.5'] -> certifi[version='>=2016.9.26']
scikit-image[version='>=0.17'] -> matplotlib-base[version='>=2.0.0'] -> certifi[version='>=2020.06.20']

Package libblas conflicts for:
numpy[version='>=1.16'] -> libcblas[version='>=3.9.0,<4.0a0'] -> libblas[version='3.9.0|3.9.0|3.9.0|3.9.0.*',build='3_openblas|6_openblas|5_openblas|4_openblas|2_openblas|1_openblas']
numpy[version='>=1.16'] -> libblas[version='>=3.9.0,<4.0a0']

Package wheel conflicts for:
python=3.8 -> pip -> wheel
pip -> wheel

[tepickering]

IIRC, python 3.9 is required for M1 support. haven't tried 3.8. however, even with 3.9 i ran into a lot of conda dependency issues with the miniforge installer for the M1. it could handle dependencies, but not dependencies of dependencies. haven't had a chance to loop back to see if there's since been a fix for that.

[stargaser]

If there is willingness to try a few things:

1. The basic workshop needs only a few packages. I think several of them are just to show that it's possible to read Excel files, for example. Perhaps: conda env create -n astropy-M1 python=3.9 astropy matplotlib pip scipy pandas followed by conda activate astropy-M1 and then pip install astroquery notebook xlrd xlwt.
2. Try the mamba package manager which is a drop-in replacement for conda.

For the workshop, we might have to make a separate set of instructions for M1; and we can have a Binder instance as a fallback. We could have troubleshooting at the Astropy booth the week after the workshop.

[isuruf]

it could handle dependencies, but not dependencies of dependencies.

This is not true. What probably happens is that you are trying to install a noarch package with a dependency that is not noarch and has not been built for the osx-arm64 platform. If you want more packages built for the osx-arm64 platform and is willing to help out, send a PR to https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/migrations/osx_arm64.txt to add whatever packages you need.

[tepickering]

that sounds like a good explanation for the root cause of what i saw. i was able to get everything installed eventually by installing the deep dependencies manually and then working upward from there. so everything i needed was built for osx-arm64, but somehow not being found. anyway, it's been a month since i worked on this so will give it another go soon.

in the meantime, i stick with my strong recommendation that users run astropy and the rest of the scientific python stack under rosetta until apple silicon is officially supported by the default conda and pip channels.

[kelle]

What has worked. I did it line-by-line:

conda create -n astropy-workshop-M1 python=3.9 
conda activate astropy-workshop-M1
conda install astropy 
conda install matplotlib
conda install scipy
conda install pandas
pip install astroquery
pip install xlrd
pip install xlwt
conda install ipython
conda install notebook
conda install cython
conda install scikit-image
conda install jupyterlab
conda install nbgitpuller
conda install asdf
conda install gwcs

And now only two things left which failed to import when I ran check_env.py:

photutils
specutils

PROGRESS!

I will start digging into how to get the utils packages installed but any hand holding advice is welcome.

[kelle]

I got more things to work! Just going to edit the above comment.

[kelle]

Now i'm running into this error when I try pip install photutils. I've tried to only paste the relevant parts:

customize UnixCCompiler
      libraries mkl_rt not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    blis_info:
      libraries blis not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    openblas_info:
      libraries openblas not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    atlas_3_10_blas_threads_info:
    Setting PTATLAS=ATLAS
      libraries tatlas not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    atlas_3_10_blas_info:
      libraries satlas not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    atlas_blas_threads_info:
    Setting PTATLAS=ATLAS
      libraries ptf77blas,ptcblas,atlas not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    atlas_blas_info:
      libraries f77blas,cblas,atlas not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    accelerate_info:
      libraries accelerate not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
    Library accelerate was not found. Ignoring
      libraries veclib not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
    Library veclib was not found. Ignoring
      FOUND:
        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

      FOUND:
        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

non-existing path in 'numpy/distutils': 'site.cfg'
    lapack_opt_info:
    lapack_mkl_info:
      libraries mkl_rt not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    openblas_lapack_info:
      libraries openblas not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    openblas_clapack_info:
      libraries openblas,lapack not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    flame_info:
      libraries flame not found in ['/Users/kelle/miniforge3/envs/astropy-workshop-M1/lib', '/usr/lib']
      NOT AVAILABLE

    atlas_3_10_threads_info:
    Setting PTATLAS=ATLAS
      libraries lapack_atlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries tatlas,tatlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries lapack_atlas not found in /usr/lib
      libraries tatlas,tatlas not found in /usr/lib
    <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
      NOT AVAILABLE

    atlas_3_10_info:
      libraries lapack_atlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries satlas,satlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries lapack_atlas not found in /usr/lib
      libraries satlas,satlas not found in /usr/lib
    <class 'numpy.distutils.system_info.atlas_3_10_info'>
      NOT AVAILABLE

    atlas_threads_info:
    Setting PTATLAS=ATLAS
      libraries lapack_atlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries ptf77blas,ptcblas,atlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries lapack_atlas not found in /usr/lib
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib
    <class 'numpy.distutils.system_info.atlas_threads_info'>
      NOT AVAILABLE

    atlas_info:
      libraries lapack_atlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries f77blas,cblas,atlas not found in /Users/kelle/miniforge3/envs/astropy-workshop-M1/lib
      libraries lapack_atlas not found in /usr/lib
      libraries f77blas,cblas,atlas not found in /usr/lib
    <class 'numpy.distutils.system_info.atlas_info'>
      NOT AVAILABLE

      FOUND:
        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

 In file included from numpy/core/src/npymath/npy_math.c:9:
    numpy/core/src/npymath/npy_math_internal.h.src:490:21: warning: incompatible pointer types passing 'npy_longdouble *' (aka 'double *') to parameter of type 'long double *' [-Wincompatible-pointer-types]
        return modfl(x, iptr);
                        ^~~~
    /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/math.h:394:52: note: passing argument to parameter here
    extern long double modfl(long double, long double *);
                                                       ^
    1 warning generated.
    llvm-ar: adding 4 object files to build/temp.macosx-11.0-arm64-3.9/libnpymath.a
    error: Command "/Users/runner/miniforge3/conda-bld/python-split_1608587737618/_build_env/bin/llvm-ar rcs build/temp.macosx-11.0-arm64-3.9/libnpymath.a build/temp.macosx-11.0-arm64-3.9/numpy/core/src/npymath/npy_math.o build/temp.macosx-11.0-arm64-3.9/build/src.macosx-11.0-arm64-3.9/numpy/core/src/npymath/ieee754.o build/temp.macosx-11.0-arm64-3.9/build/src.macosx-11.0-arm64-3.9/numpy/core/src/npymath/npy_math_complex.o build/temp.macosx-11.0-arm64-3.9/numpy/core/src/npymath/halffloat.o" failed with exit status 127
    ----------------------------------------
    ERROR: Failed building wheel for numpy
  Failed to build numpy
  ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly

I get this same error with another package I'm working on so at least there's consistency!

[larrybradley]

@kelle Those warnings are coming from trying to build numpy from source (and it's not finding the OpenBlas or ATLAS libraries). I'm not sure how you installed astropy without installing numpy though.

[larrybradley]

@kelle If indeed numpy is not installed, try this: https://github.com/numpy/numpy/issues/17807

[kelle]

@larrybradley , might be easier to chat on slack? As far as I can tell, numpy is indeed installed.

numpy 1.19.4 py39h69a04d8_2 conda-forge

[tepickering]

yeah, if you can't install photutils via conda, you'll need to pull the source and do python setup.py install. i had to do that for a few packages, but would need to pry laptop away from spouse to double-check if photutils was one of them. actually reasonably sure it was...

[tepickering]

the other packages i needed to install from source ran into the same problem when installing via pip. namely, not seeing a conda-installed dependency and trying to build it, but failing. until there are osx-arm64 binary wheels, pip will be problematic on that platform.

[kelle]

cloned photutils and then ran python setup.py install from the top level directory. Had to do this first though: pip install extensions-helpers. Trying specutils now.

[kelle]

Yep, that worked (I used pip install . instead this time) and now the setup script found all the things!

Found IPython 7.19.0
Found cython 0.29.21
Found jupyter 
Found notebook 6.1.6
Found numpy 1.19.4
Found scipy 1.6.0
Found skimage 0.18.1
Found matplotlib 3.3.3
Found pandas 1.2.0
Found bs4 4.9.3
Found keyring 
Found html5lib 1.1
Found xlwt 1.3.0
Found requests 2.25.1
Found jupyterlab 3.0.1
Found nbgitpuller 0.9.0
Found astropy 4.2
Found asdf 2.7.1
Found gwcs 0.16.1
Found photutils 1.1.0.dev99+gc9ff83db
Found specutils 1.2.dev73+g91bd46a7
Found astroquery 0.4.1

[kelle]

I've put this workflow in the Google Doc Setup for M1 Mac for Scientific Astro Programming. And I think we can close this issue. :)

[stargaser]

uh-oh, I think we forgot to add ccdproc to the workshop installation instructions! I see it in the Google doc, so does that mean you've installed it already?