Crash when restarting kernel

HansBouwmeester commented 2 years ago

Description

What steps will reproduce the problem?

When I select "restart kernel" Spyder crashes.

The crash likely relates to the fact that I cannot get the kernel to properly start in the first place always reporting this error:

"Your Python environment or installation doesn't have the matplotlib module installed or it occurred a problem importing it. Due to that, it is not possible for Spyder to create this special console for you."

Note that I do have matplotlib abd spyder-kernels installed in a Python venv:

% pip list | grep matplotlib matplotlib 3.5.1 matplotlib-inline 0.1.3

% pip list | grep kernels
spyder-kernels 2.3.0

I'm running on a Mac M1 in ARM64 mode.

Traceback

Traceback (most recent call last):
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/spyder/plugins/ipythonconsole/widgets/client.py", line 681, in _restart_thread_main
    self.shellwidget.kernel_manager.restart_kernel(
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "nest_asyncio.pyc", line 81, in run_until_complete
  File "asyncio/futures.pyc", line 201, in result
  File "asyncio/tasks.pyc", line 256, in __step
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 525, in _async_restart_kernel
    await ensure_async(self.shutdown_kernel(now=now, restart=True))
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "nest_asyncio.pyc", line 81, in run_until_complete
  File "asyncio/futures.pyc", line 201, in result
  File "asyncio/tasks.pyc", line 256, in __step
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 79, in wrapper
    raise e
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 71, in wrapper
    out = await method(self, *args, **kwargs)
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 486, in _async_shutdown_kernel
    await ensure_async(self.request_shutdown(restart=restart))
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "nest_asyncio.pyc", line 81, in run_until_complete
  File "asyncio/futures.pyc", line 201, in result
  File "asyncio/tasks.pyc", line 256, in __step
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 391, in _async_request_shutdown
    self._connect_control_socket()
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 312, in _connect_control_socket
    self._control_socket = self._create_connected_socket("control")
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/connect.py", line 605, in _create_connected_socket
    sock = self.context.socket(socket_type)
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/zmq/sugar/context.py", line 262, in socket
    raise ZMQError(ENOTSUP)
zmq.error.ZMQError: Operation not supported

Versions

Spyder version: 5.3.0
Python version: 3.9.5
Qt version: 5.15.2
PyQt5 version: 5.15.6
Operating System: Darwin 21.4.0

Dependencies

# Mandatory:
applaunchservices >=0.1.7            :  0.2.1 (OK)
atomicwrites >=1.2.0                 :  1.4.0 (OK)
chardet >=2.0.0                      :  4.0.0 (OK)
cloudpickle >=0.5.0                  :  2.0.0 (OK)
cookiecutter >=1.6.0                 :  1.7.3 (OK)
diff_match_patch >=20181111          :  20200713 (OK)
intervaltree >=3.0.2                 :  3.1.0 (OK)
IPython >=7.31.1;<8.0.0              :  7.32.0 (OK)
jedi >=0.17.2;<0.19.0                :  0.18.1 (OK)
jellyfish >=0.7                      :  0.9.0 (OK)
jsonschema >=3.2.0                   :  4.4.0 (OK)
keyring >=17.0.0                     :  23.5.0 (OK)
nbconvert >=4.0                      :  6.4.5 (OK)
numpydoc >=0.6.0                     :  1.2.1 (OK)
parso >=0.7.0;<0.9.0                 :  0.8.3 (OK)
pexpect >=4.4.0                      :  4.8.0 (OK)
pickleshare >=0.4                    :  0.7.5 (OK)
psutil >=5.3                         :  5.9.0 (OK)
pygments >=2.0                       :  2.11.2 (OK)
pylint >=2.5.0                       :  2.13.3 (OK)
pyls_spyder >=0.4.0                  :  0.4.0 (OK)
pylsp >=1.4.1;<1.5.0                 :  1.4.1 (OK)
pylsp_black >=1.2.0                  :  1.2.0 (OK)
qdarkstyle >=3.0.2;<3.1.0            :  3.0.3 (OK)
qstylizer >=0.1.10                   :  0.2.1 (OK)
qtawesome >=1.0.2                    :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0             :  5.3.0 (OK)
qtpy >=2.0.1                         :  2.0.1 (OK)
rtree >=0.9.7                        :  0.9.7 (OK)
setuptools >=49.6.0                  :  60.10.0 (OK)
sphinx >=0.6.6                       :  4.5.0 (OK)
spyder_kernels >=2.3.0;<2.4.0        :  2.3.0 (OK)
textdistance >=4.2.0                 :  4.2.2 (OK)
three_merge >=0.1.1                  :  0.1.1 (OK)
watchdog >=0.10.3                    :  2.1.7 (OK)
zmq >=17                             :  22.3.0 (OK)

# Optional:
cython >=0.21                        :  0.29.28 (OK)
matplotlib >=3.0.0                   :  3.5.1 (OK)
numpy >=1.7                          :  1.22.3 (OK)
pandas >=1.1.1                       :  1.4.1 (OK)
scipy >=0.17.0                       :  1.8.0 (OK)
sympy >=0.7.3                        :  1.10.1 (OK)

# Spyder plugins:
spyder_terminal.terminalplugin 1.2.2 :  1.2.2 (OK)

ccordoba12 commented 2 years ago

Hey @Gra55h0pper, thanks for reporting. I see that you can run %pip list, even when you we don't provide pip in as part of our app. So, how are you able to do that?

I'm asking because if you installed more packages with pip inside the app, that could have broken it and we can't simply fix that.

HansBouwmeester commented 2 years ago

@ccordoba12, I just ran pip list in the venv where I have the python interpreter installed (Python 3.10.4) and where I installed spyder-kernels also (using pip install). It then gave me the error: Your Python environment or installation doesn't have the matplotlib module installed or it occurred a problem importing it. In the process of trying to fix that I ran into the Spyder crash when selecting restart kernel.

Note that I don't use (mini)conda as my application is implemented in Google App Engine which (I believe) only supports pip with a requirements.txt file.

HansBouwmeester commented 2 years ago

@ccordoba12 although the error re: matplotlib import doesn't give any trace information I have a very strong suspicion it's caused by an Intel vs ARM architecture incompatibility. I found that when I enter arch in the Spyder terminal it reports i386 where I had expected the online installer to give me arm64. As my venv and all pip packages installed in it are arm64 this is unlikely to work.

Is there a way I can install Spyder to run native M1 (arm64)?

ccordoba12 commented 2 years ago

I just ran pip list in the venv where I have the python interpreter installed (Python 3.10.4) and where I installed spyder-kernels also (using pip install)

Ok, sorry, you didn't mention that you're using an external environment.

In the process of trying to fix that I ran into the Spyder crash when selecting restart kernel.

Ok, so does the error go away if you uninstall Matplotlib?

I found that when I enter arch in the Spyder terminal it reports i386 where I had expected the online installer to give me arm64

The installer is based on Intel packages because there aren't arm64 available ones for all Spyder dependencies, but that shouldn't be a problem, I think.

@mrclary, do you have an M1 laptop available to test this?

Is there a way I can install Spyder to run native M1 (arm64)?

Not at the moment, sorry.

mrclary commented 2 years ago

I've got an M1 iMac and will check it out...

HansBouwmeester commented 2 years ago

@mrclary Sounds good!!

So summarizing, it appears to me that if I point a standalone installed Spyder on a Mac M1 to a Python venv with pip install matplotlib executed from a arm64 venv terminal Spyder reports the matplotlib module import error. Where if I pip install matplotlib from a i386 venv (Rosetta terminal) it works fine.

mrclary commented 2 years ago

I am unable to reproduce the OP on my M1 iMac.

I also see i386 as the result of arch from Spyder's terminal console and arm64 from a Terminal, but this may be a nuance of that plugin implementation that I don't fully understand.
I created a virtual environment with pyenv and installed Python 3.10.3, spyder-kernels, and matplotlib. pip list provided below.
I have no problems opening an IPython console or selecting "Restart kernel..." from the menu or using the keyboard shortcut cmd + .; the kernel restarts promptly.

@Gra55h0pper, Perhaps you could share more details? How did you create your virtual environment? can you share your pip list for that environment? Have you tried rebuilding your virtual environment?

pip list

``` Package Version ----------------- ------- appnope 0.1.3 backcall 0.2.0 cloudpickle 2.0.0 cycler 0.11.0 debugpy 1.6.0 decorator 5.1.1 entrypoints 0.4 fonttools 4.33.3 ipykernel 6.13.0 ipython 7.33.0 jedi 0.18.1 jupyter-client 7.3.0 jupyter-core 4.10.0 kiwisolver 1.4.2 matplotlib 3.5.2 matplotlib-inline 0.1.3 nest-asyncio 1.5.5 numpy 1.22.3 packaging 21.3 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.1.0 pip 22.0.4 prompt-toolkit 3.0.29 psutil 5.9.0 ptyprocess 0.7.0 Pygments 2.12.0 pyparsing 3.0.8 python-dateutil 2.8.2 pyzmq 22.3.0 setuptools 58.1.0 six 1.16.0 spyder-kernels 2.3.0 tornado 6.1 traitlets 5.1.1 wcwidth 0.2.5 wurlitzer 3.0.2 ```

17805

ccordoba12 commented 2 years ago

To me this looks more like a firewall/VPN issue because the traceback above is related to ZMQ sockets.

HansBouwmeester commented 2 years ago

Following @mrclary's finding that it works for him I tried it again (regular arm64 shell) using a "latest everything minimal approach":

/Library/Frameworks/Python.framework/Versions/3.10/bin/python3 -m venv .venv
source .venv/bin/activate
pip install spyder-kernels
pip install matplotlib

...giving me (after pointing Spyder to .venv) the matplotlib import error (and Spyder crash when restarting the kernel).

@mrclary, I noticed the following differences between our setups:

Use of venv vs. pyenv (I believe they're not exactly the same?)
I have installed python (which I use for creating the venv) using the regular python.org macOS 64-bit universal2 installer. Could that make a difference?

mrclary commented 2 years ago

Following @mrclary's finding that it works for him I tried it again (regular arm64 shell) using a "latest everything minimal approach":
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3 -m venv .venv
source .venv/bin/activate
pip install spyder-kernels
pip install matplotlib
...giving me (after pointing Spyder to .venv) the matplotlib import error (and Spyder crash when restarting the kernel).

Okay, so at least it's reproducible. And, yes, I now see where your Python is coming from...

@mrclary, I noticed the following differences between our setups:

Use of venv vs. pyenv (I believe they're not exactly the same?)

I have installed python (which I use for creating the venv) using the regular python.org macOS 64-bit universal2 installer. Could that make a difference?

This is a good question! My bet is on the different Python installation. I have Python installed via pyenv via Homebrew. I could try installing Python from python.org and see what happens...

mrclary commented 2 years ago

Hmm... I still don't have any issues. Installed Python from python.org and created a virtual environment exactly as @Gra55h0pper specified.

However, @ccordoba12 we should probably ensure that the path selection for the Python interpreter does not resolve the symlink. I navigated to /Users/ryan/Desktop/.venv/bin/python but it was immediately replaced with /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10; to work properly, I needed to actually paste the path into the path field.

HansBouwmeester commented 2 years ago

Update! Given @mrclary's findings I tried it again but this time using Python 3.9.7 from miniconda:

% conda activate
% which python 
/Users/hansb/opt/miniconda3/bin/python
% python --version 
Python 3.9.7
% /Users/hansb/opt/miniconda3/bin/python -m venv .venv_test2
% source .venv/bin/activate
% pip install spyder-kernels
% pip install matplotlib

This works!! Spyder reports:

Python 3.9.7 (default, Sep 16 2021, 08:50:36)
Type "copyright", "credits" or "license" for more information.

IPython 7.33.0 -- An enhanced Interactive Python.

Populating the interactive namespace from numpy and matplotlib

So I'm guessing some package/link-library on my computer somehow conflicts with Spyder... (?) And using Conda somehow resolves the conflict... (?)

I'm wondering if it could be something installed through homebrew??? I have quite a few of these to support other (non-Python) development environments:

% brew list
==> Formulae
abseil          gdbm            libpng          protobuf
assimp          gettext         libpq           protobuf-c
autoconf        gflags          libsodium       pyqt
automake        giflib          libtiff         pyqt@5
boost           glib            libuv           python@3.10
brotli          glog            libyaml         python@3.9
c-ares          grpc            little-cms2     qt
ca-certificates     hunspell        lz4         qt@5
cmake           icu4c           m4          re2
cocoapods       idb-companion       md4c            readline
dbus            jasper          mpdecimal       ruby
double-conversion   jpeg            node            snappy
fb303           krb5            onnxruntime     sqlite
fbthrift        libb2           openssl@1.1     wangle
fizz            libevent        openssl@3       watchman
fmt         libffi          pcre            webp
folly           libmng          pcre2           xz
freetype        libnghttp2      postgresql      zstd

==> Casks
android-sdk zulu11

HansBouwmeester commented 2 years ago

So looking at the above brew list output I tried uninstalling qt, qt@5, pyqt and pyqt@5 guessing these could be a possible concern for interference with matplotlib. No such luck however. If I install things (without using Conda of course) the crash still occurs.

HansBouwmeester commented 2 years ago

@ccordoba12 Thinking about this, the miniconda workaround should work fine for me actually! It should allow me to pip-install -r requirements.txt everything I need (within a venv) as native M1 (arm64) packages while using Spyder. Even though the production server I use (Google App Engine) cannot use Conda (at least I'm not aware it can) that's fine as it of course doesn't need to run Spyder anyway.

So from my perspective we can close this Issue if desired. However if you'd like to debug further to understand better what's going on under the hood I'd be happy to contribute of course!! I really like Spyder and have been using it for several years now!

mrclary commented 2 years ago

Update! Given @mrclary's findings I tried it again but this time using Python 3.9.7 from miniconda: ... This works!!

Excellent!

So I'm guessing some package/link-library on my computer somehow conflicts with Spyder... (?) And using Conda somehow resolves the conflict... (?)

I don't think there is any conflict with Spyder. The issue is likely the virtual environment.

I'm wondering if it could be something installed through homebrew??? I have quite a few of these to support other (non-Python) development environments:

This is also unlikely. Homebrew, the virtual environment, your installed Python, and miniconda, all live in different places and don't interact.

When you installed Python from python.org, did you also run /Applications/Python 3.10/Install Certificates.command as recommended? This version of Python uses it's own ssl certificates, not the system ones, so installation of packages with pip may not have performed as expected. I'm just speculating here.

Also, since I could not actually reproduce the error, I cannot say for sure whether the traceback in the OP is a result of the first kernel shutting down (issue #17615) rather than the virtual environment failing to start (although that was "Socket operation on non-socket" not "Operation not supported").

@Gra55h0pper, if you're so inclined, you could run Spyder in debug mode from the Terminal.

$ /Applications/Spyder.app/Contents/MacOS/Spyder --debug-info=verbose

Then proceed to engage the "bad" virtual environment and reproduce the OP error and quit Spyder. Then post the file ~/.spyder-py3/spyder-debug.log or its contents to a comment as well as any possible output in the Terminal.

ccordoba12 commented 2 years ago

However, @ccordoba12 we should probably ensure that the path selection for the Python interpreter does not resolve the symlink. I navigated to /Users/ryan/Desktop/.venv/bin/python but it was immediately replaced with /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10; to work properly, I needed to actually paste the path into the path field.

Ok, then this could be the problem, i.e. that we're using the wrong Python for venvs.

What happens of you leave the resolved symlink? Do you get a similar error as @Gra55h0pper?

mrclary commented 2 years ago

However, @ccordoba12 we should probably ensure that the path selection for the Python interpreter does not resolve the symlink. I navigated to /Users/ryan/Desktop/.venv/bin/python but it was immediately replaced with /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10; to work properly, I needed to actually paste the path into the path field.

Ok, then this could be the problem, i.e. that we're using the wrong Python for venvs.

What happens of you leave the resolved symlink? Do you get a similar error as @Gra55h0pper?

No, I do not get a similar error. In this case the IPython console does not start up and gives the error message regarding missing spyder-kernels. Of course this is what is expected, since the resolved Python executable doesn't have an associated environment in which spyder-kernels is installed; essentially it is a "naked" Python executable.

@Gra55h0pper, can you confirm that the Python executable in your Python Interpreter preference pane is the unresolved path to the virtual environment? If it is not, will you try to force this by pasting or typing the path into the field instead of using the file navigator? If spyder-kernels is somehow installed in some "root" environment, then I can imagine that the resolved Python executable could start an IPython console but reveal other errors symptomatic of a bad environment.

HansBouwmeester commented 2 years ago

Sorry for the radio silence. Had some other things that had priority.

I have in the preferences Python interpreter section: /Users/hansb/workspace/afterburner/.venv_fail/bin/python

Also I tried uninstalling miniconda3 (just to take that variable out of the equation) and installing the 'pass/fail' environments from scratch (pass == 386 (Rosetta) terminal used for install, fail == arm64 terminal):

hansb@MacBook-Air afterburner % arch
arm64
hansb@MacBook-Air afterburner % which python3
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3
hansb@MacBook-Air afterburner % python3 -m venv .venv_fail
hansb@MacBook-Air afterburner % source .venv_fail/bin/activate
(.venv_fail) hansb@MacBook-Air afterburner % pip install spyder-kernels matplotlib

I then looked at the python executable links:

(.venv_fail) hansb@MacBook-Air afterburner % which python
/Users/hansb/workspace/afterburner/.venv_fail/bin/python
(.venv_fail) hansb@MacBook-Air afterburner % ls -l /Users/hansb/workspace/afterburner/.venv_fail/bin/python
lrwxr-xr-x  1 hansb  staff  7 May 16 10:26 /Users/hansb/workspace/afterburner/.venv_fail/bin/python -> python3
(.venv_fail) hansb@MacBook-Air afterburner % which python3
/Users/hansb/workspace/afterburner/.venv_fail/bin/python3
(.venv_fail) hansb@MacBook-Air afterburner % ls -l /Users/hansb/workspace/afterburner/.venv_fail/bin/python3
lrwxr-xr-x  1 hansb  staff  62 May 16 10:26 /Users/hansb/workspace/afterburner/.venv_fail/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.10/bin/python3

And compared these to the python executable links for the working (i386) venv:

(.venv_pass) hansb@MacBook-Air afterburner % arch
i386
(.venv_pass) hansb@MacBook-Air afterburner % which python
/Users/hansb/workspace/afterburner/.venv_pass/bin/python
(.venv_pass) hansb@MacBook-Air afterburner % ls -l /Users/hansb/workspace/afterburner/.venv_pass/bin/python
lrwxr-xr-x  1 hansb  staff  7 May 16 10:42 /Users/hansb/workspace/afterburner/.venv_pass/bin/python -> python3
(.venv_pass) hansb@MacBook-Air afterburner % which python3
/Users/hansb/workspace/afterburner/.venv_pass/bin/python3
(.venv_pass) hansb@MacBook-Air afterburner % ls -l /Users/hansb/workspace/afterburner/.venv_pass/bin/python3
lrwxr-xr-x  1 hansb  staff  62 May 16 10:42 /Users/hansb/workspace/afterburner/.venv_pass/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.10/bin/python3

Seems to be the same.

HansBouwmeester commented 2 years ago

I just noticed a difference. The pip install commands tell me:

for .venv_pass

Collecting spyder-kernels
  Using cached spyder_kernels-2.3.0-py2.py3-none-any.whl (71 kB)
Collecting matplotlib
  Using cached matplotlib-3.5.2-cp310-cp310-macosx_10_9_x86_64.whl (7.3 MB)

for .venv_fail

Collecting spyder-kernels
  Using cached spyder_kernels-2.3.0-py2.py3-none-any.whl (71 kB)
Collecting matplotlib
  Using cached matplotlib-3.5.2-cp310-cp310-macosx_11_0_arm64.whl (7.2 MB)

... but this would be as expected though?

mrclary commented 2 years ago

@Gra55h0pper, I suppose I've misunderstood your setup. How do you work with both arm64 and i386 (Rosetta) terminals? I don't understand this.

HansBouwmeester commented 2 years ago

@mrclary, see for example: https://apple.stackexchange.com/questions/428768/on-apple-m1-with-rosetta-how-to-open-entire-terminal-iterm-in-x86-64-architec

1) Go to Finder > Applications and find your Terminal 2) Right-Click the App and Duplicate it and rename it "Terminal i386" 3) Right-Click "Terminal i386" > Get Info > Enable Open using Rosetta 4) Click to Open the Terminal, type arch to verify it says i386 now. 5) Right-Click the Terminal i386 in your Dock and click "Keep in Dock" for future use.

So if I (pip)-install spyder-kernels + matplotlib in a venv created in such a Rosetta i386 terminal it works. If I run the same install commands in a regular arm64 terminal it doesn't (matplotlib import error).

mrclary commented 2 years ago

@mrclary, see for example: https://apple.stackexchange.com/questions/428768/on-apple-m1-with-rosetta-how-to-open-entire-terminal-iterm-in-x86-64-architec

Go to Finder > Applications and find your Terminal

Right-Click the App and Duplicate it and rename it "Terminal i386"

Right-Click "Terminal i386" > Get Info > Enable Open using Rosetta

Click to Open the Terminal, type arch to verify it says i386 now.

Right-Click the Terminal i386 in your Dock and click "Keep in Dock" for future use.

Ahhh. I missed this from the very beginning, I'm sorry.

So if I (pip)-install spyder-kernels + matplotlib in a venv created in such a Rosetta i386 terminal it works. If I run the same install commands in a regular arm64 terminal it doesn't (matplotlib import error).

Okay, I will try to replicate the issue again. Just to be clear the steps are:

create a venv using the i386 terminal
install spyder-kernels and matplotlib into this venv using the i386 terminal
Everything works.

and

create a venv using the i386 terminal
install spyder-kernels and matplotlib into this venv using the arm64 terminal
IPython console fails.

If this is correct, then I suspect that pip is checking package compatibility against package version requirements only, not platform, but requesting the platform version based on the current Terminal arch, so that you end up with a matplotlib arm64 version rather than the x86_64 version. If a package has any architecture specific bindings (C, C++, Fortran, etc.), then this will break.

HansBouwmeester commented 2 years ago

Nr. 1 in the second case should read: create a venv using the arm64 terminal. (But I think that's what you meant to say).

HansBouwmeester commented 2 years ago

If this is correct, then I suspect that pip is checking package compatibility against package version requirements only, not platform, but requesting the platform version based on the current Terminal arch, so that you end up with a matplotlib arm64 version rather than the x86_64 version. If a package has any architecture specific bindings (C, C++, Fortran, etc.), then this will break.

Interesting thought! So I tried (in the non-working (i.e. arm64) terminal+venv):

arch -x86_64 pip install --force-reinstall matplotlib

Spyder then seems to work in the sense that the iPython console does not give the error anymore and import matplotlib works and is able to generate plots!!!

The only weird thing is that now the terminal pane inside Spyder (which I hardly ever use to be honest) doesn't seem to initialize properly as it just shows a completely white background (and nothing else).

HansBouwmeester commented 2 years ago

...of course I then should have immediately tried to add the same treatment for spyder-kernels:

arch -x86_64 pip install --force-reinstall spyder-kernels

Works!!!

Now I'm curious if you're able to replicate my findings or whether I'd still have something unusual happening on my system...

HansBouwmeester commented 2 years ago

So summarizing, I just verified that the following minimal install procedure using venv works for me using a standalone Spyder-install on an Apple Mac M1 with external arm64 Python packages (other than spyder-kernels and matplotlib).

All commands executed from a regular MacOS M1 arm64 terminal:

python3 -m venv .venv
source .venv/bin/activate
arch -x86_64 pip install spyder-kernels matplotlib
pip install -r whatever.txt

(So simple, I cannot believe I didn't think of trying this before...)

mrclary commented 2 years ago

~~Actually, I think there may be an issue with Spyder here...~~

So, with Spyder 5.3.0 macOS app and the Python installed from python.org (universal2 installer) I've done the following:

Using an arm64 Terminal (default) to create arm64env virtual environment and install spyder-kernels and matplotlib.
- Spyder works fine and IPython consoles startup and restart just fine with arm64env
- Importing numpy in Spyder's IPython console gives the following error. There appears to be an architecture issue.
- However, importing numpy from an IPython console started from an arm64 Terminal in the arm64env environment has no such error.
  numpy traceback

In [1]: import numpy
Traceback (most recent call last):

  File "/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray

  File "/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides

  File "/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (

ImportError: dlopen(/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): tried: '/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/var/folders/sc/1rrvv5750vx9bdrq5cbktybh0000gn/T/ipykernel_54554/2172125874.py", line 1, in <cell line: 1>
    import numpy

  File "/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/__init__.py", line 144, in <module>
    from . import core

  File "/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)

ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.10 from "/Users/ryan/arm64env/bin/python"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): tried: '/Users/ryan/arm64env/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

Using an i386 Terminal to create i386env virtual environment and install spyder-kernels and matplotlib.
- Spyder works fine and IPython consoles startup and restart just fine with the i386env environment
- Importing numpy works just fine
- Importing numpy from an IPython console started from an i386 Terminal in the i386env environment has no errors.

So, I think I need to investigate a matrix of settings: Spyder = {standalone app, local source}; environment type = {venv (python.org), conda, pyenv}; terminal arch = {i386, arm64}

mrclary commented 2 years ago

This does not appear to be a Spyder issue, but there are some issues related to arm64 architecture for PyQt5.

The latest Spyder artifact, which is built on x86_64 architecture, appears to work fine and its "Same as Spyder" IPython console also appears to work fine. There only seems to be potential issues with external pip environments.
Conda flavored environments seem to be okay; they have wheels for PyQt5 5.15.* for the arm64 architecure and everything seems to install properly.
Pip environments do not appear to have wheels for any version of PyQt5 for the arm64 architecture. Lacking wheels, pip downloads the source code and attempts to build locally, but fails. I have not been able to find an acceptable solution for this after scouring the internet.
Wheels do exist for PyQt5 for the x86-64 architecture, so these can be used if a universal2 python is used. However, this requires, at minimum, prefixing all python related commands with arch -x86_64. To avoid prefixing all commands, one could install rosetta2 from the command line, duplicate the Terminal app and dedicate it to opening with rosetta and use this Terminal for all x86_64 commands. I leave it to the reader to decide which is more convenient for their purpose. Note that Python.org python is a universal2 installation, but pyenv python versions are not universal2 by default. If one is using pyenv, PYTHON_CONFIGURE_OPTS must include --enable-universalsdk and --with-universal-archs=universal2 in order to install the universal2 python.

While I was unable to reproduce the OP, I believe that the core issue was related to architecture incompatibility in the virtual environment, and not Spyder.

@spyder-ide/core-developers, I don't believe there is anything for us to do here, unless someone can exert influence with PyQt5 developers to upload arm64 wheels to PyPi.

@ccordoba12 do you feel comfortable closing this issue?

ccordoba12 commented 2 years ago

Thanks for the thorough research @mrclary! After reading your post, I also agree there's nothing we can do about it on our side, so I'm going to close this issue.

@Gra55h0pper, I think it's better for you to use Conda and Conda-forge, which has now full support for the arm64 architecture on Mac.

mrclary commented 2 years ago

@Gra55h0pper, I realize that you are working on a native arm64 platform but doing development for x86_64 platforms. This may become a more common issue as M1 Macs gain market share among Mac users. At present, as @ccordoba12 says, for developing for arm64, conda-forge is best as it has the best arm64 support. However for developing on arm64 but for x86_64, your solution of using the x86_64 version of the Python universal2 installation is a good one. However, for other users that come across this issue, the key take-away is to remember to not mix platform environments and always prefix python commands with arch -x86_64. Hopefully, more universal2 wheels will become available to keep pace with the M1 adoption and obviate the need to create dedicated -x86_64 environments for development.

spyder-ide / spyder