save workspace fail when using dotty-dict

[nikfio]

Description

What steps will reproduce the problem?

tried to save a workspace in a .spydata file. Failed. Various data types inside, str pandas bool and so on

Traceback

Traceback (most recent call last):
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 124, in comm_channel_manager
    yield
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 225, in _get_call_return_value
    return super(KernelComm, self)._get_call_return_value(
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder_kernels\comms\commbase.py", line 435, in _get_call_return_value
    self._wait_reply(call_id, call_name, timeout)
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 247, in _wait_reply
    self._wait(got_reply, self._sig_got_reply, timeout_msg, timeout)
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 284, in _wait
    raise TimeoutError(timeout_msg)
TimeoutError: Timeout while waiting for {'26296de15bb946f0abd26eb94a5262f5': (False, <bound method ShellWidget.remote_set_cwd of <spyder.plugins.ipythonconsole.widgets.shell.ShellWidget object at 0x000002C108E5C0D0>>), '99d50ffab3eb4820838f0012bafc6780': (False, <bound method ShellWidget.remote_set_cwd of <spyder.plugins.ipythonconsole.widgets.shell.ShellWidget object at 0x000002C108E5C0D0>>), 'dae056a2ea9b40098096d47b3fdfcd11': (True, None)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\variableexplorer\widgets\main_widget.py", line 192, in <lambda>
    triggered=lambda x: self.save_data(),
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\variableexplorer\widgets\main_widget.py", line 493, in save_data
    nsb.save_data()
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\variableexplorer\widgets\namespacebrowser.py", line 260, in save_data
    error_message = self.shellwidget.save_namespace(self.filename)
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\widgets\namespacebrowser.py", line 176, in save_namespace
    return self.call_kernel(
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder_kernels\comms\commbase.py", line 557, in __call__
    return self._comms_wrapper._get_call_return_value(
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 225, in _get_call_return_value
    return super(KernelComm, self)._get_call_return_value(
  File "C:\python_envs\Systems_analysis_n_prediction\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 128, in comm_channel_manager
    self._comms[comm_id]['comm']._send_channel = (
KeyError: 'db1ffefa11df11ed83509c305bd3aab8'

Versions

• Spyder version: 5.3.0
• Python version: 3.9.13
• Qt version: 5.15.3
• PyQt5 version: 5.15.4
• Operating System: Windows 10

Dependencies

# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  5.0.0 (OK)
cloudpickle >=0.5.0           :  2.1.0 (OK)
cookiecutter >=1.6.0          :  2.1.1 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.31.1;<8.0.0       :  7.33.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jellyfish >=0.7               :  0.9.0 (OK)
jsonschema >=3.2.0            :  4.6.1 (OK)
keyring >=17.0.0              :  23.6.0 (OK)
nbconvert >=4.0               :  6.5.0 (OK)
numpydoc >=0.6.0              :  1.2.1 (OK)
paramiko >=2.4.0              :  2.11.0 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.9.1 (OK)
pygments >=2.0                :  2.12.0 (OK)
pylint >=2.5.0                :  2.14.4 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.4.1;<1.5.0          :  1.4.1 (OK)
pylsp_black >=1.2.0           :  1.2.1 (OK)
qdarkstyle >=3.0.2;<3.1.0     :  3.0.3 (OK)
qstylizer >=0.1.10            :  0.2.1 (OK)
qtawesome >=1.0.2             :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0      :  5.3.1 (OK)
qtpy >=2.0.1                  :  2.1.0 (OK)
rtree >=0.9.7                 :  1.0.0 (OK)
setuptools >=49.6.0           :  63.1.0 (OK)
sphinx >=0.6.6                :  5.0.2 (OK)
spyder_kernels >=2.3.0;<2.4.0 :  2.3.1 (OK)
textdistance >=4.2.0          :  4.3.0 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.9 (OK)
zmq >=17                      :  23.2.0 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=3.0.0            :  3.5.2 (OK)
numpy >=1.7                   :  1.23.0 (OK)
pandas >=1.1.1                :  1.4.3 (OK)
scipy >=0.17.0                :  None (NOK)
sympy >=0.7.3                 :  None (NOK)

[dalthviz]

Hi @nikfio thank you for the feedback! Could it be possible for you to update to our latest release (Spyder 5.3.2) and check again? Also, could you share some example code that enables us to reproduce this in our side? Any new info is greatly apprecited Let us know!

[nikfio]

Hello @dalthviz, thank you for your reply. I have updated Spyder to version 5.3.2 as ou suggested, but the problem keeps occurring. I'll try to expand. Basically I'm testing my code with breakpoints, at breakpoint reached I have this situation

Then I click the 'save data as' button just above the 'variable explorer' and the saving operation crashes.

The crash shows a sort of error traceback on the IPdb console.

Let me know If you need more information.

Thanks Nick

[nikfio]

@dalthviz I'm doing a deeper investigation on this issue.

Here's a larger screen of the IPnb console with the save data crash occurred

You can see a lot of rows like this

File "C:\python_envs\Systems_analysis_n_prediction\lib\site-packages\dotty_dict\dotty_dict.py", line 89 in __getattr__

I'm currently using a library called dotty dict (https://dotty-dict.readthedocs.io/en/latest/dotty_dict.html). It really seems like that instancing a dotty object

from dotty_dict import dotty test_dotty = dotty()

makes spyder save data operation going nuts.

Avoiding this instance, data saving goes ok. It just gives out this pop up which I think it is expected (?)

Indeed the self object is not saved but all the other variables are fine.

Hope this debugging could better help you.

Nick

[dalthviz]

Hi @nikfio tank you for the new info! I was able to reproduce this by running the dotty example code. I got the same message in the console before it restarts:

Windows fatal exception: stack overflow

Main thread:
Thread 0x000017e4 (most recent call first):
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\selectors.py", line 315 in _select
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\selectors.py", line 324 in select
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\asyncio\base_events.py", line 1860 in _run_once
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\asyncio\base_events.py", line 600 in run_forever
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\site-packages\tornado\platform\asyncio.py", line 215 in start
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\site-packages\ipykernel\kernelapp.py", line 712 in start
  File "e:\acer\documentos\spyder\spyder\external-deps\spyder-kernels\spyder_kernels\console\start.py", line 334 in main
  File "e:\acer\documentos\spyder\spyder\external-deps\spyder-kernels\spyder_kernels\console\__main__.py", line 24 in <module>
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\runpy.py", line 86 in _run_code
  File "C:\Users\dalth\anaconda3\envs\spyder-cf\lib\runpy.py", line 196 in _run_module_as_main

Restarting kernel...

Also, as I think you mention, checking the Exclude unsupported data types before using the saving functionality prevents the error (although it also prevents saving some variables).

Also, checking the traceback this seems similar to the one at https://github.com/spyder-ide/spyder/issues/18930

Seems like having a custom __getattr__ doesn't play well with jupyter kernels (used by Spyder but also Jupyter notebooks and other jupyter-based tooling).

Maybe we should ensure saving just data that is supported (even when the option Exclude unsupported data types isn't checked) @ccordoba12 ?

[ccordoba12]

Maybe we should ensure saving just data that is supported (even when the option Exclude unsupported data types isn't checked) @ccordoba12 ?

Yeah, I think that's a good idea. We could also try to switch to Dill or Cloudpickle as our libraries to save the current workspace because they do it in a more robust way.

[ncbloch]

Seems like having a custom __getattr__ doesn't play well with jupyter kernels (used by Spyder but also Jupyter notebooks and other jupyter-based tooling).

I looked into this a bit when i submitted #18905. It's not that custom __getattr__ implementations don't work with Spyder. They just have to be really robustly implemented in order to prevent infinite recursion as described in #18930. And they also need to be well behaved (e.g. throwing AttributeError on non-existent attributes). It seems like Spyder will sometimes query (possibly non-existent) attributes like some_var.shape or similar on session variables, so the getattr implementation has to be ready for for this (e.g. not throwing an AttributeError will cause problems, if I remember correctly). Maybe something similar is happening when saving the workspace?

[dalthviz]

Thanks @ncbloch for the feedback! Checking a little bit I think to save dict variables and variables in general we try to pickle them which maybe could be causing the issue here, but yep what you said makes a lot of sense :+1: at the end Spyder for the Variable Explorer and IPython Console functionality tries to get the attributtes from the variables in a way that could cause a not so robust __getattr__ implementations to end up raising a RecursionError (which with the default interpreter causes a stackoverflow on Windows as we described at #18930)