spyder-ide / spyder

Official repository for Spyder - The Scientific Python Development Environment
https://www.spyder-ide.org
MIT License
8.22k stars 1.59k forks source link

Crash in IPython console when running Numpy/Scipy code after Spyder is started from the Start menu #21749

Closed ba05 closed 7 months ago

ba05 commented 7 months ago

Some commands crash the kernel when run in the Spyder iPython console vs the Python CLI. See https://github.com/ipython/ipython/issues/14266

ccordoba12 commented 7 months ago

Hey @ba05, thanks for reporting. Could you confirm that running that code outside Spyder works as expected?

ba05 commented 7 months ago

Yes I can confirm. Works fine running from a Python CLI inside an environment without Spyder installed.

ccordoba12 commented 7 months ago

(Copying original comment from IPython to have it here for quick reference)


To reproduce issue:

mamba create --name testEnv python=3.12 spyder scipy numpy

At the python console:

import numpy as np
import scipy.signal as signal
np.cov([1,2,3],[1.1, 2.1, 3]) # works, no error
signal.savgol_filter([1,2,3], window_length=2, polyorder=1) # works, no error

The above results in "Windows fatal exception: code 0xc06d007e" when run on the Spyder ipython console.

Packages:

``` Name Version Build Channel alabaster 0.7.16 pyhd8ed1ab_0 conda-forge arrow 1.3.0 pyhd8ed1ab_0 conda-forge astroid 3.0.2 py312h2e8e312_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge atomicwrites 1.4.1 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pyh71513ae_0 conda-forge autopep8 2.0.4 pyhd8ed1ab_0 conda-forge babel 2.14.0 pyhd8ed1ab_0 conda-forge bcrypt 4.1.2 py312hfccd98a_0 conda-forge beautifulsoup4 4.12.3 pyha770c72_0 conda-forge binaryornot 0.4.4 py_1 conda-forge black 24.1.1 py312h2e8e312_0 conda-forge bleach 6.1.0 pyhd8ed1ab_0 conda-forge brotli-python 1.1.0 py312h53d5487_1 conda-forge bzip2 1.0.8 hcfcfb64_5 conda-forge ca-certificates 2023.11.17 h56e8100_0 conda-forge certifi 2023.11.17 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py312he70551f_0 conda-forge chardet 5.2.0 py312h2e8e312_1 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge click 8.1.7 win_pyh7428d3b_0 conda-forge cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.2.1 pyhd8ed1ab_0 conda-forge cookiecutter 2.5.0 pyhca7485f_0 conda-forge cryptography 42.0.1 py312he4c2ea2_0 conda-forge debugpy 1.8.0 py312h53d5487_1 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge diff-match-patch 20230430 pyhd8ed1ab_0 conda-forge dill 0.3.8 pyhd8ed1ab_0 conda-forge docstring-to-markdown 0.13 pyhd8ed1ab_0 conda-forge docutils 0.20.1 py312h2e8e312_3 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge flake8 6.1.0 pyhd8ed1ab_0 conda-forge gettext 0.21.1 h5728263_0 conda-forge glib 2.78.3 h12be248_0 conda-forge glib-tools 2.78.3 h12be248_0 conda-forge gst-plugins-base 1.22.9 h001b923_0 conda-forge gstreamer 1.22.9 hb4038d2_0 conda-forge icu 73.2 h63175ca_0 conda-forge idna 3.6 pyhd8ed1ab_0 conda-forge imagesize 1.4.1 pyhd8ed1ab_0 conda-forge importlib-metadata 7.0.1 pyha770c72_0 conda-forge importlib_metadata 7.0.1 hd8ed1ab_0 conda-forge importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge inflection 0.5.1 pyh9f0ad1d_0 conda-forge intel-openmp 2024.0.0 h57928b3_49841 conda-forge intervaltree 3.1.0 pyhd8ed1ab_1 conda-forge ipykernel 6.29.0 pyha63f2e9_0 conda-forge ipython 8.20.0 pyh7428d3b_0 conda-forge isort 5.13.2 pyhd8ed1ab_0 conda-forge jaraco.classes 3.3.0 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jellyfish 1.0.3 py312hfccd98a_0 conda-forge jinja2 3.1.3 pyhd8ed1ab_0 conda-forge jsonschema 4.21.1 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge jupyter_client 8.6.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.1 py312h2e8e312_0 conda-forge jupyterlab_pygments 0.3.0 pyhd8ed1ab_0 conda-forge keyring 24.3.0 py312h2e8e312_0 conda-forge krb5 1.21.2 heb0366b_0 conda-forge libblas 3.9.0 21_win64_mkl conda-forge libcblas 3.9.0 21_win64_mkl conda-forge libclang 15.0.7 default_hde6756a_4 conda-forge libclang13 15.0.7 default_h85b4d89_4 conda-forge libexpat 2.5.0 h63175ca_1 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libglib 2.78.3 h16e383f_0 conda-forge libhwloc 2.9.3 default_haede6df_1009 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge libjpeg-turbo 3.0.0 hcfcfb64_1 conda-forge liblapack 3.9.0 21_win64_mkl conda-forge libogg 1.3.4 h8ffe710_1 conda-forge libpng 1.6.39 h19919ed_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libspatialindex 1.9.3 h39d44d4_4 conda-forge libsqlite 3.44.2 hcfcfb64_0 conda-forge libvorbis 1.3.7 h0e60522_0 conda-forge libwebp 1.3.2 hcfcfb64_1 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxml2 2.12.4 hc3477c8_1 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.4 py312he70551f_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mccabe 0.7.0 pyhd8ed1ab_0 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mistune 3.0.2 pyhd8ed1ab_0 conda-forge mkl 2024.0.0 h66d3029_49657 conda-forge more-itertools 10.2.0 pyhd8ed1ab_0 conda-forge mypy_extensions 1.0.0 pyha770c72_0 conda-forge nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert 7.14.2 pyhd8ed1ab_0 conda-forge nbconvert-core 7.14.2 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 7.14.2 pyhd8ed1ab_0 conda-forge nbformat 5.9.2 pyhd8ed1ab_0 conda-forge nest-asyncio 1.6.0 pyhd8ed1ab_0 conda-forge numpy 1.26.3 py312h8753938_0 conda-forge numpydoc 1.6.0 pyhd8ed1ab_0 conda-forge openssl 3.2.0 hcfcfb64_1 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandoc 3.1.3 h57928b3_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge paramiko 3.4.0 pyhd8ed1ab_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge pathspec 0.12.1 pyhd8ed1ab_0 conda-forge pcre2 10.42 h17e33f8_0 conda-forge pexpect 4.9.0 pyhd8ed1ab_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pip 23.3.2 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge platformdirs 4.1.0 pyhd8ed1ab_0 conda-forge pluggy 1.4.0 pyhd8ed1ab_0 conda-forge ply 3.11 py_1 conda-forge prompt-toolkit 3.0.42 pyha770c72_0 conda-forge psutil 5.9.8 py312he70551f_0 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pycodestyle 2.11.1 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pydocstyle 6.3.0 pyhd8ed1ab_0 conda-forge pyflakes 3.1.0 pyhd8ed1ab_0 conda-forge pygments 2.17.2 pyhd8ed1ab_0 conda-forge pylint 3.0.3 pyhd8ed1ab_0 conda-forge pylint-venv 3.0.3 pyhd8ed1ab_0 conda-forge pyls-spyder 0.4.0 pyhd8ed1ab_0 conda-forge pynacl 1.5.0 py312hc560f31_3 conda-forge pyqt 5.15.9 py312he09f080_5 conda-forge pyqt5-sip 12.12.2 py312h53d5487_5 conda-forge pyqtwebengine 5.15.9 py312hca0710b_5 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge python 3.12.1 h2628c8c_1_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.19.1 pyhd8ed1ab_0 conda-forge python-lsp-black 2.0.0 pyhd8ed1ab_0 conda-forge python-lsp-jsonrpc 1.1.2 pyhd8ed1ab_0 conda-forge python-lsp-server 1.9.0 pyhd8ed1ab_0 conda-forge python-lsp-server-base 1.9.0 pyhd8ed1ab_0 conda-forge python-slugify 8.0.2 pyhd8ed1ab_0 conda-forge python_abi 3.12 4_cp312 conda-forge pytoolconfig 1.2.5 pyhd8ed1ab_0 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pywin32 306 py312h53d5487_2 conda-forge pywin32-ctypes 0.2.2 py312h2e8e312_1 conda-forge pyyaml 6.0.1 py312he70551f_1 conda-forge pyzmq 25.1.2 py312h1ac6f91_0 conda-forge qdarkstyle 3.2.3 pyhd8ed1ab_0 conda-forge qstylizer 0.2.2 pyhd8ed1ab_0 conda-forge qt-main 5.15.8 h9e85ed6_18 conda-forge qt-webengine 5.15.8 h4bf5c4e_4 conda-forge qtawesome 1.3.0 pyh9208f05_0 conda-forge qtconsole 5.5.1 pyhd8ed1ab_0 conda-forge qtconsole-base 5.5.1 pyha770c72_0 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge referencing 0.32.1 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rich 13.7.0 pyhd8ed1ab_0 conda-forge rope 1.12.0 pyhd8ed1ab_0 conda-forge rpds-py 0.17.1 py312hfccd98a_0 conda-forge rtree 1.2.0 py312h72b5f30_0 conda-forge scipy 1.12.0 py312h8753938_2 conda-forge setuptools 69.0.3 pyhd8ed1ab_0 conda-forge sip 6.7.12 py312h53d5487_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snowballstemmer 2.2.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sphinx 7.2.6 pyhd8ed1ab_0 conda-forge sphinxcontrib-applehelp 1.0.8 pyhd8ed1ab_0 conda-forge sphinxcontrib-devhelp 1.0.6 pyhd8ed1ab_0 conda-forge sphinxcontrib-htmlhelp 2.0.5 pyhd8ed1ab_0 conda-forge sphinxcontrib-jsmath 1.0.1 pyhd8ed1ab_0 conda-forge sphinxcontrib-qthelp 1.0.7 pyhd8ed1ab_0 conda-forge sphinxcontrib-serializinghtml 1.1.10 pyhd8ed1ab_0 conda-forge spyder 5.5.0 py312h2e8e312_3 conda-forge spyder-kernels 2.5.0 win_pyh7428d3b_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tbb 2021.11.0 h91493d7_1 conda-forge text-unidecode 1.3 pyhd8ed1ab_1 conda-forge textdistance 4.5.0 pyhd8ed1ab_0 conda-forge three-merge 0.1.1 pyh9f0ad1d_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5226925_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tomlkit 0.12.3 pyha770c72_0 conda-forge tornado 6.3.3 py312he70551f_1 conda-forge traitlets 5.14.1 pyhd8ed1ab_0 conda-forge types-python-dateutil 2.8.19.20240106 pyhd8ed1ab_0 conda-forge typing-extensions 4.9.0 hd8ed1ab_0 conda-forge typing_extensions 4.9.0 pyha770c72_0 conda-forge tzdata 2023d h0c530f3_0 conda-forge ucrt 10.0.22621.0 h57928b3_0 conda-forge ujson 5.9.0 py312h53d5487_0 conda-forge urllib3 2.1.0 pyhd8ed1ab_0 conda-forge vc 14.3 hcf57466_18 conda-forge vc14_runtime 14.38.33130 h82b7239_18 conda-forge vs2015_runtime 14.38.33130 hcb4865c_18 conda-forge watchdog 3.0.0 py312h2e8e312_1 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge whatthepatch 1.0.5 pyhd8ed1ab_0 conda-forge wheel 0.42.0 pyhd8ed1ab_0 conda-forge win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge yapf 0.40.1 pyhd8ed1ab_0 conda-forge zeromq 4.3.5 h63175ca_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zstd 1.5.5 h12be248_0 conda-forge ```

Originally posted by @ba05 in https://github.com/ipython/ipython/issues/14266#issuecomment-1915179626

ccordoba12 commented 7 months ago

Yes I can confirm. Works fine running from a Python CLI inside an environment without Spyder installed.

So, the problem persists if you install Spyder in the environment and run that code outside of it?

ba05 commented 7 months ago

The problem never appears in the Python CLI with Spyder installed in the environment.

I did some more testing on my end. Running the following code...:

import numpy as np
import scipy.signal as signal
np.cov([1,2,3],[1.1, 2.1, 3])
signal.savgol_filter([1,2,3], window_length=2, polyorder=1)

with the following configs:

1) works: Running from CLI with Spyder installed environment. 2) works: Running from Spyder IPython (launching Spyder by typing 'spyder' at mamba command prompt). 3) fails: Running from Spyder IPython (launching Spyder from Windows Start menu shortcut).

The environments above were created using mamba. Now, if I create a mamba environment with Spyder and then:

pip uninstall numpy scipy
pip install numpy scipy

3) works: Running from Spyder IPython (launching Spyder from Windows Start menu shortcut).

Error seen when failure occurs:

Windows fatal exception: code 0xc06d007e

Main thread:
Current thread 0x00007bcc (most recent call first):
  File "C:\Users\AronBW\AppData\Local\miniforge3\envs\testEnv3\Lib\site-packages\numpy\lib\function_base.py", line 2747 in cov
  File "C:\Users\AronBW\AppData\Local\Temp\ipykernel_29652\2177909138.py", line 1 in <module>```

Restarting kernel...

Why does the pip uninstall / install fix things? And why is there no error when Spyder is launched from the mamba prompt versus the Windows start menu shortcut?

evanhorn commented 7 months ago

I am seeing the same issue with scipy.optimize.curve_fit. I found that I could run the code as a Jupyter notebook in the same environment without problem, but after converting it to a script the code crashed the kernel in Spyder. I was also able to run the script using WSL (Ubuntu 22.04) - that is obviously a different environment, but verified to me that the problem was not with the code.

After finding this report, I confirmed that I could run my code after starting Spyder via the Anaconda PowerShell prompt, and after reinstalling scipy with pip, the code does run in Spyder when started from the Start menu.

ccordoba12 commented 7 months ago

The problem never appears in the Python CLI with Spyder installed in the environment.

I did some more testing on my end. Running the following code...:

  1. works: Running from CLI with Spyder installed environment.
  2. works: Running from Spyder IPython (launching Spyder by typing 'spyder' at mamba command prompt).
  3. fails: Running from Spyder IPython (launching Spyder from Windows Start menu shortcut).

Ok, thanks for the additional info @ba05, that's really valuable. This could be related to updates to a package called menuinst.

@mrclary, could you check if you are able to reproduce this with menuinst 1.4 and 2.0? Thanks!

mrclary commented 7 months ago

@ba05 @evanhorn, can you confirm the menuinst version in your base conda environment?

$ mamba list -n base menusint
ba05 commented 7 months ago

@ba05 @evanhorn, can you confirm the menuinst version in your base conda environment?

$ mamba list -n base menusint

Here you go (Note your commend had a typo, menusint vs menuinst)

# Name                    Version                   Build  Channel
menuinst                  1.4.19          py310h5588dad_1    conda-forge
mrclary commented 7 months ago

Just to provide information regarding menuinst:

I will check to see if existing Spyder versions do anything strange for menuinst >=2.0. At least for the present issue, I don't think it is the result of menuinst since @ba05 reported version <2.0.

mrclary commented 7 months ago

This is the error I get:

In [1]: import numpy as np
   ...: import scipy.signal as signal
   ...: np.cov([1,2,3],[1.1, 2.1, 3]) # works, no error
   ...: signal.savgol_filter([1,2,3], window_length=2, polyorder=1) # works, no error

Windows fatal exception: code 0xc06d007e

Main thread:
Current thread 0x000020b0 (most recent call first):
  File "C:\Users\rclary\.conda\envs\issue-21771\Lib\site-packages\numpy\lib\function_base.py", line 2747 in cov
  File "C:\Users\rclary\AppData\Local\Temp\ipykernel_10528\1129720154.py", line 3 in <module>

Restarting kernel...

In [1]:

The shortcut target (for me) is

C:\Users\rclary\AppData\Local\miniforge3\pythonw.exe C:\Users\rclary\AppData\Local\miniforge3\cwp.py C:\Users\rclary\.conda\envs\issue-21771 C:\Users\rclary\.conda\envs\issue-21771\pythonw.exe C:\Users\rclary\.conda\envs\issue-21771\Scripts\spyder-script.py

You can run this from a vanilla cmd terminal and reproduce the error (be sure to replace your correct paths).

At this point, I don't think it is menuinst, at least not directly. I've done some preliminary investigation and gotten the shortcut to work in some circumstance but not others and don't yet know what to make of it. Is it the environment activation? Is it an issue with pythonw vs python? Is it a conda or python version? I'm not sure yet. But I'll report back.

I've reproduced the error with our latest master Spyder and menuinst >=2.0, so this issue will be important to figure out.

evanhorn commented 7 months ago

It appears the discussion has moved passed suspecting menuinst, but in case it's helpful, I have menuinst == 1.4.20 installed in my base conda environment.

ccordoba12 commented 7 months ago

@evanhorn or @ba05, could you downgrade (if possible) to menuinst 1.4.17, install Spyder and Numpy in a new environment and try again?

ccordoba12 commented 7 months ago

This is a comment in an issue reported last week that could help to solve this conundrum:


Not sure if this is a bug... For sure I would not try to repeat this, but somehow after first failing with broken environments but then finally succeeding in updating to conda python 3.10 and latest Spyder 5.4.3 I found myself in the state where Spyder would launch perfectly from my environment, but would crash immediately if launched from the Start Menu. This all on Windows 10. The error message in Event Log pointed me to KERNELBASE.DLL, but that turned out to be a red herring.

If I copied the start menu command and typed it into an Anaconda terminal, then Spyder would launch. If I copied that start menu command into a regular terminal, then Spyder would immediately die with the same KERNELBASE.DLL error reported to the Event log. After hacking around in PDB, I found that the environment being created by CWP.PY was not quite the same environment that did work, specifically it was missing the ....\my-env\bin folder, which has in it LIBIOMP5MD.DLL

I changed CWP.PY to add the term join(prefix, "bin"), to the new_paths and, amazing, Spyder now opens perfectly from the startup menu.

As I say, I don't know if I had a broken environment somehow, but if anyone else is reporting similar mysterious errors on launch then it might be appropriate to update cwp.py for everyone.

Originally posted by @Tunneller in https://github.com/spyder-ide/spyder/issues/21760#issue-2115442380

ba05 commented 7 months ago

@evanhorn or @ba05, could you downgrade (if possible) to menuinst 1.4.17, install Spyder in a new environment and try again?

Looks like I am unable to downgrade due to not allowing Python 3.12 (which I am running):

warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
Could not solve for environment specs
The following packages are incompatible
└─ menuinst 1.4.17**  is installable with the potential options
   ├─ menuinst 1.4.17 would require
   │  └─ python >=3.6,<3.7.0a0 , which can be installed;
   ├─ menuinst 1.4.17 would require
   │  └─ python >=3.7,<3.8.0a0 , which can be installed;
   ├─ menuinst 1.4.17 would require
   │  └─ python >=3.8,<3.9.0a0 , which can be installed;
   └─ menuinst 1.4.17 would require
      └─ python >=3.9,<3.10.0a0 , which can be installed.
mrclary commented 7 months ago

@Tunneller @ccordoba12, I can corroborate @Tunneller's workaround in #21760. I'll investigate further, but there may actually be a bug in conda's shell activate as well...

mrclary commented 7 months ago

Okay, there is not a bug with conda's shell activate, but cwp.py is certainly missing the additional %PREFIX%\bin path in PATH. This is likely a bug, since I believe that cwp.py is intended to replicate the conda run -p mechanism, which does correctly add that path to PATH.

Nevertheless, this issue is affecting Spyder's IPython console because Spyder does not activate the conda environment if the console environment is the same as Spyder's runtime environment. The simplest workaround for now is to create a separate conda environment and use that for IPython consoles.

We can fix this problem for 5.x and 6.x moving forward by ensuring that all conda environments are activated in KernelSpec.argv, not just external ones.

ccordoba12 commented 7 months ago

We decided to fix this issue in our next version (5.5.2), to be released in a couple of weeks. In the meantime, please start Spyder from the command line after activating your conda env.