Spyder kernel crash while saving workspace to .spydata file

diegobersanetti commented 3 years ago

Description

What steps will reproduce the problem?

Spyder kernel crashes and restart while trying to save the workspace in a .spydata file. This always happens at the same point (given the same workspace).

The .pickle file is empty; the 0000 .npy file is 144 byte, and the crash happens while finalizing the 0008 file (47.2/48 MB).

Platform: Ubuntu Linux 20.04.2, conda 4.9.2, Spyder 4.2.1, Python 3.7.9

Traceback

  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/comms/kernelcomm.py", line 92, in comm_channel_manager
    yield
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/comms/kernelcomm.py", line 166, in _get_call_return_value
    call_dict, call_data, comm_id)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder_kernels/comms/commbase.py", line 432, in _get_call_return_value
    self._wait_reply(call_id, call_name, timeout)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/comms/kernelcomm.py", line 187, in _wait_reply
    self._wait(got_reply, self._sig_got_reply, timeout_msg, timeout)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/comms/kernelcomm.py", line 224, in _wait
    raise TimeoutError(timeout_msg)
TimeoutError: Timeout while waiting for {'affaf6ebc2474cbcb9c6bd4369f09f6d': (True, None)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/variableexplorer/widgets/namespacebrowser.py", line 511, in save_data
    error_message = self.shellwidget.save_namespace(self.filename)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/widgets/namespacebrowser.py", line 177, in save_namespace
    timeout=CALL_KERNEL_TIMEOUT).save_namespace(filename)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder_kernels/comms/commbase.py", line 555, in __call__
    call_dict, call_data, self._comm_id)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/comms/kernelcomm.py", line 166, in _get_call_return_value
    call_dict, call_data, comm_id)
  File "/home/blind/Software/anaconda3/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/blind/Software/anaconda3/lib/python3.7/site-packages/spyder/plugins/ipythonconsole/comms/kernelcomm.py", line 96, in comm_channel_manager
    self.kernel_client.shell_channel)
KeyError: '8b7848aa745c11eba536bf81c31c5f06'

Versions

Spyder version: 4.2.1
Python version: 3.7.9
Qt version: 5.9.6
PyQt5 version: 5.9.2
Operating System: Linux 5.8.0-43-generic

Dependencies


# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=7.6.0                 :  7.20.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  22.0.1 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.8.0 (OK)
pygments >=2.0                  :  2.7.4 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.1 (OK)
qtconsole >=5.0.1               :  5.0.2 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  52.0.0.post20210125 (OK)
sphinx >=0.6.6                  :  3.4.3 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.10.1 (OK)
textdistance >=4.2.0            :  4.2.1 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  1.0.2 (OK)
xdg >=0.26                      :  0.27 (OK)
zmq >=17                        :  20.0.0 (OK)

# Optional:
cython >=0.21                   :  None (NOK)
matplotlib >=2.0.0              :  3.3.4 (OK)
numpy >=1.7                     :  1.19.2 (OK)
pandas >=1.1.1                  :  1.2.2 (OK)
scipy >=0.17.0                  :  1.6.0 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

steff456 commented 3 years ago

Hi @diegobersanetti,

Is this happening with a certain pickle file or with multiple ones? If is only one file, can you please upload it here to see if we can track this issue?

ccordoba12 commented 3 years ago

@steff456, I think the problem is a variable is too large and so we can't retrieve its value in order to save it.

So I think the fix is to catch that TimeoutError and show a message if that's the case.

diegobersanetti commented 3 years ago

Hello, I made some tests and the very same routine works with smaller data sets, so I guess it could be an out-of-memory issue or something similar to that; if it could be of interest I can try however to provide some more data about it.

ccordoba12 commented 3 years ago

so I guess it could be an out-of-memory issue or something similar to that

Thanks @diegobersanetti, that's exactly my guess.

Right now we wait 30 secs to retrieve a variable from the kernel and then give up, which is required to save your current workspace.

So we have two options to solve this problem:

Show a warning message telling users it was not possible to save their workspace because it's too big.
Show a warning telling them they could wait longer (1 min, 3 min, 5 min) to see if we manage to retrieve and save its contents.

What option would you prefer?

ccordoba12 commented 3 years ago

I think we can't do this without a timeout because this operation blocks Spyder, right @impact27?

impact27 commented 3 years ago

It does but it does not have to. We could just use a callback in save_namespace instead. same thing for load_data. get_value should probably be left alone.

spyder-ide / spyder