search

nteract / hydrogen

:atom: Run code interactively, inspect data, and plot. All the power of Jupyter kernels, inside your favorite text editor.
https://nteract.gitbooks.io/hydrogen/
MIT License
3.92k stars 334 forks source link

import error for numpy on Apple M1 with python 3.8 #2147

Open hycis opened 2 years ago

hycis commented 2 years ago

Description

On Apple M1, the default architecture is arm64, and import works ok on Terminal with python 3.8. But when I import NumPy on Hydrogen, it returns the following error.

ImportError: dlopen(/Users/joewu/Library/Python/3.8/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): 
tried: '/Users/joewu/Library/Python/3.8/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' 
(mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), 
'/usr/local/lib/_multiarray_umath.cpython-38-darwin.so' (no such file),
 '/usr/lib/_multiarray_umath.cpython-38-darwin.so' (no such file)

This error can be reproduce if I set the arch == x86 in the Terminal and import numpy in terminal. If I downgrade the python3.8 to python3.7, this error goes off, because python 3.7 defaults uses x86 compiled packages. The easiest way to fix this error in Hydrogen is to set the arch env to 'arm64'. I try to look into the hydrogen code and env and can't find a way to set the default arch in Hydrogen to 'arm64'. I wonder how this bug can be fixed.

hycis commented 2 years ago

I wonder how hydrogen set the env for arch by default

hycis commented 2 years ago

hydrogen default runs on x86 arch, this arch will return numpy import error on python 3.8

import platform
platform.machine() # returns x86_64

if I import numpy on terminal that runs arch arm64, it will run without error. if I set arch from arm64 to x86 and reimport numpy on terminal, it will return the same exact error as when I run it on hydrogen. So I am pretty sure the bug lies with default arch on hydrogen being set as x86 when runs with python 3.8. I wonder how do we change the default arch in hydrogen to arm64

hycis commented 2 years ago 
joewu@Joes-Mac-mini ~ % jupyter --version
Selected Jupyter core packages...
IPython          : 8.1.1
ipykernel        : 6.9.2
ipywidgets       : 7.7.0
jupyter_client   : 7.1.2
jupyter_core     : 4.9.2
jupyter_server   : not installed
jupyterlab       : not installed
nbclient         : 0.5.13
nbconvert        : 6.4.4
nbformat         : 5.2.0
notebook         : 6.4.10
qtconsole        : 5.2.2
traitlets        : 5.1.1
python   :3.8.9
alesubbiah commented 2 years ago

Hello! Has this bug been resolved? I am having the same issue importing Numpy within ATOM on an M1 Pro machine. I am using a virtual environment, that works on VSCode, but does not work on ATOM.

When I run the platform.machine() on VSCode though, I get 'arm64', whereas in Hydrogen I get 'x86_64', so this seems to be the issue.

Numpy also runs from the terminal. This is a list of the packages that pip list returns (cut the bottom half for space).

-------------------- -------
appnope              0.1.3
argon2-cffi          21.3.0
argon2-cffi-bindings 21.2.0
asttokens            2.0.5
attrs                21.4.0
backcall             0.2.0
beautifulsoup4       4.11.1
bleach               5.0.0
cffi                 1.15.0
debugpy              1.6.0
decorator            5.1.1
defusedxml           0.7.1
entrypoints          0.4
executing            0.8.3
fastjsonschema       2.15.3
gensim               4.1.2
importlib-resources  5.7.0
ipykernel            6.13.0
ipython              8.2.0
ipython-genutils     0.2.0
ipywidgets           7.7.0
jedi                 0.18.1
Jinja2               3.1.1
joblib               1.1.0
jsonschema           4.4.0
jupyter              1.0.0
jupyter-client       7.2.2
jupyter-console      6.4.3
jupyter-core         4.9.2
jupyterlab-pygments  0.2.1
jupyterlab-widgets   1.1.0
MarkupSafe           2.1.1
matplotlib-inline    0.1.3
mistune              0.8.4
nbclient             0.6.0
nbconvert            6.5.0
nbformat             5.3.0
nest-asyncio         1.5.5
notebook             6.4.10
numpy                1.22.3
packaging            21.3
pandas               1.4.2
pandocfilters        1.5.0
parso                0.8.3
pexpect              4.8.0
pickleshare          0.7.5
pip                  22.0.4
...

The error I get is: 

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
File ~/wpp/lib/python3.8/site-packages/numpy/core/__init__.py:23, in <module>
     22 try:
---> 23     from . import multiarray
     24 except ImportError as exc:

File ~/wpp/lib/python3.8/site-packages/numpy/core/multiarray.py:10, in <module>
      9 import functools
---> 10 from . import overrides
     11 from . import _multiarray_umath

File ~/wpp/lib/python3.8/site-packages/numpy/core/overrides.py:6, in <module>
      4 import os
----> 6 from numpy.core._multiarray_umath import (
      7     add_docstring, implement_array_function, _get_implementing_args)
      8 from numpy.compat._inspect import getargspec

ImportError: dlopen(/Users/ale.subbiah/wpp/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): tried: '/Users/ale.subbiah/wpp/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 import numpy as np

File ~/wpp/lib/python3.8/site-packages/numpy/__init__.py:144, in <module>
    141 # Allow distributors to run custom init code
    142 from . import _distributor_init
--> 144 from . import core
    145 from .core import *
    146 from . import compat

File ~/wpp/lib/python3.8/site-packages/numpy/core/__init__.py:49, in <module>
     25     import sys
     26     msg = """
     27 
     28 IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
   (...)
     47 """ % (sys.version_info[0], sys.version_info[1], sys.executable,
     48         __version__, exc)
---> 49     raise ImportError(msg)
     50 finally:
     51     for envkey in env_added:

ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/Users/ale.subbiah/wpp/bin/python"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/ale.subbiah/wpp/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): tried: '/Users/ale.subbiah/wpp/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

Again, seems to be something to do with the architecture When I run pip list from within atom: numpy (1.22.3) seems to be installed correctly in the desired virtual env as you can see from above. My python version is also 3.8.9.

sys.path in ATOM Hydrogen is pointing to the same sys.path on VSCode: 

['/Users/ale.subbiah',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
 '',
 '/Users/ale.subbiah/wpp/lib/python3.8/site-packages']

Except that in VSCode it also seems to be pointing to the Python inside my /Library as shown below: 

['/',
 '/Users/ale.subbiah/.vscode/extensions/ms-toolsai.jupyter-2022.3.1000901801/pythonFiles',
 '/Users/ale.subbiah/.vscode/extensions/ms-toolsai.jupyter-2022.3.1000901801/pythonFiles/lib/python',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
 '',
 '/Users/ale.subbiah/wpp/lib/python3.8/site-packages',
 '/Users/ale.subbiah/Library/Python/3.8/lib/python/site-packages']

Any help on solving this would be greatly appreciated!

hycis commented 2 years ago

it's a bug with hydrogen/atom apparently.

alesubbiah commented 2 years ago

Thanks for responding! :) Can confirm it works perfectly with Python 3.9.2