Memory leak while having spyder run too much stuff or for too long.

leet0rz commented 3 years ago

Description

What steps will reproduce the problem?

Just having a couple of windows open, leave the program running for 1-2 days, sometimes hours and it will slow down real bad to a point pressing enter for new lines takes a couple of seconds. Obviously unacceptable, back to VScode until you guys fix this.

Versions

Spyder version: 4.2.1
Python version: 3.7.9
Qt version: 5.12.10
PyQt5 version: 5.12.3
Operating System: Windows 10

Dependencies


# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree                    :  None (OK)
IPython >=7.6.0                 :  7.19.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  21.5.0 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.7.3 (OK)
pygments >=2.0                  :  2.7.3 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.2 (OK)
qtconsole >=5.0.1               :  5.0.1 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  51.0.0 (OK)
sphinx >=0.6.6                  :  3.3.1 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.10.1 (OK)
textdistance >=4.2.0            :  4.2.0 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog                        :  1.0.2 (OK)
zmq >=17                        :  20.0.0 (OK)

# Optional:
cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.3 (OK)
numpy >=1.7                     :  1.19.3 (OK)
pandas >=1.1.1                  :  1.1.5 (OK)
scipy >=0.17.0                  :  1.5.4 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

steff456 commented 3 years ago

Hi @leet0rz,

Can you share with me more information about what things you had opened in Spyder to see if we can reproduce this issue?

leet0rz commented 3 years ago

Window 1: 40 lines of code Window 2: Regular notes Window 3: 12 lines of code Window 4: 14 lines of code

Just basic, simple code that I did not even run, nothing major. It'll also happen if I just open a regular txt file with notes of mine. I am sure the same would happen if I open absolutely nothing and kept it running for a few days. If you do this and hold enter creating new lines it'll start lagging and chopping its way while making new lines.

ccordoba12 commented 3 years ago

By windows you mean editor tabs @leet0rz? Also, could you post a screenshot of your current setup so we can take a look at it? Thanks!

leet0rz commented 3 years ago

Yeah windows as in tabs. Here is a screenshot currently, 4 tabs open: https://i.imgur.com/6iYb1My.png

On Thu, Feb 25, 2021 at 6:30 PM Carlos Cordoba notifications@github.com wrote:

By windows you mean editor tabs @leet0rz https://github.com/leet0rz? Also, could you post a screenshot of your current setup so we can take a look at it? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/spyder-ide/spyder/issues/14792#issuecomment-786072712, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6S3GD3AJ7V6H53MWU5YGDTA2CLDANCNFSM4X66D2KQ .

MuellerSeb commented 2 years ago

I got the very same problem.

ccordoba12 commented 2 years ago

@MuellerSeb, most of these problems should be fixed in our latest version (5.2.1).

MuellerSeb commented 2 years ago

I got 5.2.1 installed. Only extension I have installed is the spyder-terminal. Checking my open processes I found that this command:

/usr/bin/python3 -m spyder_kernels.console -f /home.....

fills up my RAM with 5MB / sec while doing nothing. This is 300MB / min. After half an hour with spyder in the background, and other stuff open my system begins to stutter, since there are 10GB of ram allocated by this process.

So I think, this is not solved.

MuellerSeb commented 2 years ago

I tried:

pip uninstall spyder-terminal
pip install -I --no-cache-dir spyder

But still, I open spyder and can watch the memory being filled by the same process.

ccordoba12 commented 2 years ago

There has to be something else going on not related to Spyder because I've had a Spyder instance open for two weeks on Linux and I haven't seen any memory leak.

Given this line

/usr/bin/python3 -m spyder_kernels.console -f /home.....

it seems you installed Spyder with pip in your system Python. If that's the case, you probably broke it and now you have to reinstall your distro, or at least remove everything under /usr/local/lib/pythonX.X. That's because you're mixing regular packages and system ones, which usually breaks other important packages as well.

After that, please try to use Spyder with Anaconda/Miniconda, or to install it with pip in a virtualenv.

peendebak commented 2 years ago

@leet0rz @ccordoba12 I can reproduce what seems to be spyder memory leak on windows. Spyder version is 5.2.2. I did the following:

Determine which type of objects occur most in memory by using import gc; gc.get_objects() and then some processing
On my system (after running for a while):

memory report: <class 'cell'>: 590686 <class 'tuple'>: 503049 <class 'function'>: 387187 <class 'dict'>: 365367 <class 'list'>: 273560

I get the objects of type class 'cell' with the following code:


import gc
from typing import Optional
def get_objects_of_type(object_type : str, limit:Optional[int] =None):
objects = [ obj  for obj in gc.get_objects() if str(object_type)==str(type(obj))]
if limit is not None:
    objects=objects[:limit]
return objects

oo=get_objects_of_type("<class 'cell'>", limit=None)

4. There are many objects of this type, but just printing the first few gives a strong indication that spyder is involved:

for o in oo[:20]: print(o) <cell at 0x000001C394836850: SpyderKernel object at 0x000001C305619FA0> <cell at 0x000001C394847AF0: SpyderKernel object at 0x000001C305619FA0> <cell at 0x000001C39484D3D0: SpyderKernel object at 0x000001C305619FA0> <cell at 0x000001C39484FD30: SpyderKernel object at 0x000001C305619FA0> <cell at 0x000001C3948555E0: SpyderKernel object at 0x000001C305619FA0> ...

A difference with the issue as described above is that for me the system does not really slow down, but at some point I get out of memory errors.

peendebak commented 2 years ago

@ccordoba12 This still happens with Spyder 5.3 (Windows).

ccordoba12 commented 2 years ago

According to the Python docs,

“Cell” objects are used to implement variables referenced by multiple scopes.

and

This de-referencing of the cell object requires support from the generated byte-code; these are not automatically de-referenced when accessed

So it seems a very low level feature that doesn't happen directly in Spyder (hence not something we can fix here) but in spyder-kernels. My guess is that that's related to the way IPython handles its several namespaces and probably not something that can be fixed, sorry

leet0rz commented 2 years ago

@peendebak Yeah, I just started using vscode instead. Tons of other editors use python and do not have these memory leak issues meanwhile Spyder does for some reason, and it should not have these issue in this day and age. Maybe it's base is broken with some broken code, I dont know but other editors manage to work without leaks.

ccordoba12 commented 2 years ago

@leet0rz, most leaks should be fixed now in our latest version (5.3.0). If you're using an older one, please update and try again.

peendebak commented 2 years ago

@leet0rz @ccordoba12 I used heapy to track down some of the objects. I can now reproduce the increasing number of cell items in plain python as well. In spyder itself (or just a plain spyder-kernel) I see no additional memory leakage (tested spyder 5.3 on python 3.9).

I will see whether I can pinpoint the issue, but it indeed does not seem related (directly) to spyder.

stevetracvc commented 2 years ago

@ccordoba12 I'm using Spyder 5.3.0 on Ubuntu, installed via conda, and have something similar. I was training a few different models for a couple days straight (different consoles, using TF/Keras which print the progress to the console and graphs to the plots pane). It filled my RAM and started eating away at my swap space. When the trainings were done, I closed all of the consoles which recovered about half of my memory, but the spyder process itself was still using 55GB RAM (yes, fifty five). I opened the internal console to poke around and try to find what might be using the memory, but accidentally killed it :laughing: don't call spy.window.destroy()

Sometimes, running three or four consoles that are all training will end up slowing down Spyder to an unusable state, even if only half of my RAM is in use. Switching between tabs is painfully slow (ie, a minute), the progress stops updating in the console pane, and this is usually precipitated by me trying to open a new console while those others are running. (This is a recurring issue, so I usually just restart spyder every few days).

I've tried running in debug mode, but the logs grow at tens of MB a minute so I couldn't let it run long enough to really see what's going on.

Any suggestions for how I can find what's eating the memory? It may be Spyder related since I had killed all of the spyder_kernels.console processes (except the new console that was created after closing them all) And your quote was about de-referencing, not removing, which I'm interpreting as C pointer de-referencing, no? And if it is an issue with cell objects, they do have a dealloc, but I don't know anything about how any of that works. https://github.com/python/cpython/blob/main/Objects/cellobject.c#L161

ccordoba12 commented 2 years ago

Sometimes, running three or four consoles that are all training will end up slowing down Spyder to an unusable state, even if only half of my RAM is in use

This could have to do with the Variable Explorer consuming too much memory. To check that, please go to the menu

Tools > Preferences > Plugins

and disable it

imagen

stevetracvc commented 2 years ago

Awesome thanks for the suggestion, I'll let you know!

stevetracvc commented 2 years ago

@ccordoba12 the variable explorer didn't seem to change anything. I played a bit with what peendebak said above, looking at all the objects gc can list. It seems like a huge part of the problem is the large amount of text that my script generates (I turned off the progress bars and that seems to have made a huge impact). However, doing something as simple as

while True:
    print("all work and no play makes Jack a dull boy")

does NOT replicate it, so it must be something more complicated.

Even after closing the consoles, something else still must be referencing these objects (spyder?). For example, there are still references to matplotlib pngs generated during my training:

{'fig': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x06\xaa\x00\x00\x02\x99\x08\x06\x00\x00\x00\xc0\xb2z\xeb\x00\x00\x009tEXtSoftware\x00Matplotlib version3.5.1, https://matplotlib.org/\xd8a\xf2\xbd\x00\x00\x00\tpHYs\x00\x00\x0b\x13\x00\x00\x0b\x13\x01\x00\x9a\x9c\x18\x00\x00\xf1\xfeIDATx\x9c\xec\xddwX\x14\xd7\xfb>\xfe\x9b&\x02\x0b,M\x01A\xb0\xa3\xd8b\x8b\x1d5\xf6\xde+\x8a\x1a[\xd4h\x12\x8d\x1a5v\x8d\xfd\x9d\xa8I\x8cQ\xb1w\xa3\xc6\xae1\xd6X\xa2&j\xb0b\x01\x15\xb0\x00\x02\x02\xd2\x9f\xdf\x1f\xfe\x98/\x0b\xdb\xb0\xad\xc9\xe7~]\x17\x97\xee\x9c)g\xce\x9c9s\

...

'fmt': 'image/png', 'fwidth': 1706, 'fheight': 665, '_blink_flag': False, '_qpix_orig': <PyQt5.QtGui.QPixmap object at 0x7f2b6c0456d0>, '_qpix_scaled': <PyQt5.QtGui.QPixmap object at 0x7f2b6c063740>}

Any thoughts on how I could release these references? After closing the console, or removing all plots, I'd expect them to be freed from memory. For example, gc.get_referrers(gc.get_objects()[-95]) (where gc.get_objects()[-95] is a png from matplotlib) lists <spyder.plugins.plots.widgets.figurebrowser.FigureCanvas object at 0x7fe79495cdc0>

ccordoba12 commented 2 years ago

For example, there are still references to matplotlib pngs generated during my training

I think you hit the nail on the head! That probably means that you're experiencing the bug reported on issue #12847. Please read it for some workarounds.

mhdadk commented 2 years ago

Hi @ccordoba12, I'm facing a similar problem, but I think I found its cause and a possible solution. My problem is that Spyder slowly starts using up more and more of my RAM even though I'm not using Spyder for anything and it is left idle. Eventually, my entire PC starts to lag due to the excessive use of RAM. I tried the workaround in #12847, but this did not work. Specifically, I added matplotlib.interactive(False) to my code, but this didn't help. However, I think I've found the reason that this happens. Before I describe this in detail, here are my specs:

Spyder version: 5.3.1 None
Python version: 3.10.5 64-bit
Qt version: 5.15.3
PyQt5 version: 5.15.4
Operating System: Linux 5.13.0-28-generic

I'm also running Ubuntu 20.04.4, and I installed miniconda and not the full anaconda.

I first installed Spyder in a new conda environment using the conda install -c conda-forge spyder command. I then ran spyder --reset to make sure that Spyder is in its default settings. At this point, I do not get the memory leak problem. However, as soon as I go to Tools --> Preferences --> IPython console --> Graphics --> Graphics backend, and when I change "Backend" from "Inline" to "Automatic", my problem starts to happen. I'm not sure what the exact reason is internally though.

What solved my problem, however, was as follows:

I uninstalled miniconda completely (by deleting the miniconda3 directory in /home).
I installed the full anaconda (and not miniconda).
I created a new conda environment with a specific version of Python using the command conda create --name spyder-env python=3.8.13.
I then installed Spyder using the command conda install spyder. This installed version 5.1.5 of Spyder and not version 5.3.1 as above.
I then went to Tools --> Preferences --> IPython console --> Graphics --> Graphics backend, and changed "Backend" from "Inline" to "Automatic". However, this time, so far, my problem has not appeared. I will update this comment in case it does.

UPDATE 1 Unfortunately, while installing Spyder 5.1.5 solved the memory leak problem, debug mode no longer works properly, as described in #17314. The solution to this, as described here, is to install Spyder 5.2.2. I did so using the command conda install -c conda-forge spyder=5.2.2. Luckily, debug mode has started to work properly again. Also, so far, the memory leak problem has not showed up either. Again, I will update this comment in case anything changes. It seems that this memory leak problem may have been introduced somewhere between versions 5.2.2 and 5.3.1 of Spyder.

UPDATE 2 It turns out that installing Spyder 5.2.2 via conda-forge should be done after installing Spyder 5.1.5 via the anaconda channel. When I created a new environment and then ran the command conda install -c conda-forge spyder=5.2.2 directly, and then tried to start Spyder using

$ spyder

I kept getting this error

Traceback (most recent call last):
  File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/qtpy/QtWebEngineWidgets.py", line 21, in <module>
    from PyQt5.QtWebEngineWidgets import QWebEnginePage
ModuleNotFoundError: No module named 'PyQt5.QtWebEngineWidgets'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mahmedta/anaconda3/envs/openglue/bin/spyder", line 11, in <module>
    sys.exit(main())
  File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/spyder/app/start.py", line 247, in main
    from spyder.app import mainwindow
  File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/spyder/app/mainwindow.py", line 57, in <module>
    from qtpy import QtWebEngineWidgets  # analysis:ignore
  File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/qtpy/QtWebEngineWidgets.py", line 28, in <module>
    from PyQt5.QtWebKitWidgets import QWebPage as QWebEnginePage
ModuleNotFoundError: No module named 'PyQt5.QtWebKitWidgets'

To get around this error, I removed the current environment and created a new one. This time, I first ran

$ conda install spyder

which installed Spyder 5.1.5 from the anaconda channel. Then, I ran

$ conda install -c conda-forge spyder=5.2.2

which installed version 5.2.2 of Spyder from the conda-forge channel. When I did this, Spyder started without the PyQt5 error above.

stevetracvc commented 2 years ago

For example, there are still references to matplotlib pngs generated during my training

I think you hit the nail on the head! That probably means that you're experiencing the bug reported on issue #12847. Please read it for some workarounds.

@ccordoba12 Thanks, but unfortunately it's definitely deeper than this. I haven't had the time to dig enough into the cause, but here's some more info...

If I run a hyperparameter optimizing script that either:

Does the actual (keras) model training in the same process as the console
I use multiprocessing.Process() to train one model in its own process and then kill that process after the training

option 1 definitely has a faster memory leak than option 2 (though both do have an issue). With option 2, I can see overall system memory usage increasing during the training, then dropping off when the training process is killed. After enough cycles though, I can see overall usage is slowly creeping up.

When I have some free time, I'll try to figure out what's occupying most of the memory.

ccordoba12 commented 2 years ago

@stevetracvc, that doesn't seem related to Spyder. I mean, perhaps memory is not released with option 1. because the objects used to do the hyperparameter optimization are retained in memory after it finishes.

Could you try to run option 1. in a Jupyter notebook to check if the memory leak also shows up there? Since Spyder and Jupyter use the same architecture to run code, that would imply that your problem is not caused by Spyder.

kian25 commented 1 year ago

I'm having this issue too, I left it open running a few threads printing off data from a Binance endpoint and now running a simple print command on a different cell takes 10-20 seconds to show up in the console. If I restart, everything works fine

pearlmam commented 1 year ago

For some reason the RAM usage just slowly climbs with Spyder running with the Graphics Backend set to "tkinter". I changed the Graphics Backend from "tkinter" to "Qt5" and this solved my memory leak problem. This is with Spyder 5.3.3

kian25 commented 1 year ago

thanks for the suggestion but unfortunately it didn't seem to work for me. I was on inline before and now it feels that it's slightly slower.

I wonder if it's just a performance thing. There are 3 threads running and when they are off the individual cell commands become responsive. I migrated this code from vscode and it had no issues there, maybe it's the way spider checks for new commands, when we hit run on individual threads while threads are active, and not actual lag. Because the code itself runs fine, just new commands in different cells take 10-15 seconds

kian25 commented 1 year ago

It's really a shame, the UI for this IDE, the individual cells, consolidated output, undock and customization and organizing features but the delays just keep getting worst.

Going to go back to Vscode grudgingly, but I'll save this file as is if there is any way I can help

3 threads running and then in a new cell print('something') and it'll take forever to actually do something.

Only thing I changed is changed New consoles to the working directory of the current console

Spyder 5.42 Python 3.9.15 64-bit | Qt 5.15.2 | PyQt5 5.15.7 | Windows 10

ccordoba12 commented 1 year ago

For some reason the RAM usage just slowly climbs with Spyder running with the Graphics Backend set to "tkinter". I changed the Graphics Backend from "tkinter" to "Qt5" and this solved my memory leak problem. This is with Spyder 5.3.3

@pearlmam, that's fixed in our latest version (5.4.2). So, please update.

Going to go back to Vscode grudgingly, but I'll save this file as is if there is any way I can help

@kian25, please upload it somewhere so we can take a look at it. That way we can try to reproduce your problem on our side and see how to fix it.

costantinoai commented 8 months ago

I'm encountering a persistent memory problem, similar to one reported years ago in this issue, where the pylsp process progressively consumes all the RAM (64 GB) when using Spyder IDE, specifically in environments with fastai installed. This problem occurs on two separate machines, and does not affect other environments.

Symptoms:

The pylsp process is connected to127.0.0.x and consumes significant RAM, leading to system instability and Spyder crashes.
Leaving the pylsp process running slows down execution, functions lookup, and in general the whole IDE. Eventually, it will eat the whole RAM and lead to system or IDE sudden crash.
Terminating the pylsp process frees up RAM and temporarily resolves the issue but disables important features like linting and completion. Furthermore, Spyder restarts the process within seconds, reintroducing the problem.
Disabling LSP from Spyder's settings circumvents the RAM issue but at the cost of disabling essential coding features.

This issue forces me to use Spyder solely for code editing and basic execution, while I resort to running scripts for model training and testing via the terminal to avoid crashes and data loss.

Temporary Workaround: Currently, my solution is to disable the LSP within Spyder settings. This action, while effective at preventing memory leaks, significantly limits my ability to use Spyder for a full development workflow, especially for features like variable and class definition lookups.

My environment has been exported with conda list --export > environment_packages.txt and can be found here. OS is Ubuntu 22.04.

Any idea if this is related to the issues mentioned above, or yet another (separate) memory leak issue with Spyder?

spyder-ide / spyder