Poor handling of large dictionaries in variable explorer

[nc011]

Issue Report Checklist

• [x ] Searched the issues page for similar reports
• [ x] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• [ x] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• [x ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

After attempting to open a very large dictionary (with multiple large keys) in variable explorer and closing the "Error: too large" prompt, the console and variable explorer become unresponsive. The spinning "loading" icon in the top right of the screen continues to spin.

Spyder remains responsive, e.g. can open menu items, but the console and variable explorer hang.

It seems like, although the error is being thrown, Spyder is still actually attempting to open the dictionary rather than quitting that process. Perhaps it is attempting to access the other keys in the dictionary?

What steps reproduce the problem?

1. Create/load a very large dictionary (e.g. see example code)
2. Attempt to open the dictionary in variable explorer
3. Close the "too large" prompt 4a. Attempt to open another variable - no response from variable explorer, OR 4b. Create a new variable in the console - note that the console either doesn't process this or, if it does, the variable does not appear in the variable explorer.

Example code:

import numpy as np

var = np.zeros((5,100000000))
dic={}
dic["key1"] = var
dic["key2"] = var
dic["key3"] = var
dic["key4"] = var

What is the expected output? What do you see instead?

By clicking close on the error dialogue pop-up, it is expected that Spyder would promptly return to a useable state. Both the console and variable explorer should be responsive and the spinning loading icon should also disappear.

As an additional improvement, it would be nice if Spyder handled this situation more elegantly than just throwing an error. For very large dictionaries such as this, it would be great if Spyder showed the key names and sizes - even if it can't open the key for editing.

Versions

• Spyder version: 4.2.0
• Python version: 3.8.5 64-bit
• Qt version: 5.9.7
• PyQt version: 5.9.2
• Operating System name/version: Windows 10

Dependencies

# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=4.0                   :  7.19.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.7.2 (OK)
pygments >=2.0                  :  2.7.4 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.1;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.1.1             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.1 (OK)
qtconsole >=4.7.7               :  4.7.7 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  51.3.3.post20210118 (OK)
sphinx >=0.6.6                  :  3.4.3 (OK)
spyder_kernels >=1.10.0;<1.11.0 :  1.10.1 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  0.10.4 (OK)
zmq >=17                        :  20.0.0 (OK)

# Optional:
cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.2 (OK)
numpy >=1.7                     :  1.19.2 (OK)
pandas >=1.1.1                  :  1.2.0 (OK)
scipy >=0.17.0                  :  1.5.2 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

[steff456]

Hi @Caseyb87,

I'm able to reproduce this issue and thanks for the feedback, we'll have this in mind for our UI/UX enhancements. We can probably add pympler to the variable explorer to try to get the size in memory of the objects so we know if they are going to open or not.

Thanks for reporting!

[nc011]

Hi @steff456 ,

I think this is more than just an enhancement. It's a bug. Regardless of whether you can get the size or not, the console and explorer should not become unresponsive. It totally breaks the user's workflow.

[cesimeone]

Hello @steff456, I am also having this problem while running Spyder 5.2.2 and Python 3.9. I have plenty of RAM (36 GB), not sure why this is happening. Very disruptive.