Open pkubryk opened 3 years ago
Something very odd happened when I tried to debug in mplfinance.plot on the iteration that usually fails -> it went past it. It gives me the impression there is a race condition somewhere.
Edit : this is consistently reproducible on my case. going in debug and doing a step by step works. After that, the plot window remains opened ( although blank ) for each subsequent chart,. The save is still happening. It then fails on bitmap number 740
The are several items of interest in your code and in your report of the issue:
**Fail to allocate bitmap
" error. I would presume that there would be some kind of a Python Traceback as a result of that exception. Please provide the complete traceback, and/or other entire error messages (that may have been output before the application ended) in order *to be able to know exactly where in the code "**Fail to allocate bitmap
" is happening*.After that, the plot window remains opened
" ... since you are doing savefig
no plot window should be opening.mplfinance.plot()
does not return anything, unless you set kwarg returnfig=True
, so there is no point to setting fig = mplfinance.plot(...).
Also, if you do set returnfig=True
then it will return both the Figure and all Axes objects. (Click here for more information on gaining access to mplfinance's Figure and Axes objects).savefig
nor closefig
in mpf.plot()
. Rather set returnfig=True
and then use Figure.savefig()
to save the plot, followed by deleting the Figure. The code might look something like this:
for ix in df.index:
...
fig, axlist = mpf.plot(data, type='candle',volume=True,title=titleString,returnfig=True)
fig.savefig(detailsPath+titleString+'.jpg',dpi=150)
# notice in `fig.savefig()` that `fname` kwarg is ***not*** necessary, just pass file name as first argument
del fig
If you need more help, please provide the information requested. Thanks for reporting this issue. Seems an interesting/challenging problem to solve. All the best. --Daniel
Daniel,
Thank you for you detailed answer. Below some feedback
1- I could not find any. I tried several things to try to catch the exception or skip it for that particular item without success, ( see the except at the end of the code snippet). That makes me think it is a very low level error. Is there a better way to capture it ?
2- You are absolutely right, the plot does not remain open unless I run the code in debug. As mentioned, in debug I manage to get past the item causing problem ( without crashing the process), but after that the plot windows remain open. see screenshots below
I actually figured out that no matter where I place the breakpoint , and no matter which iteration of the loop I am in, it stops closing the figure if I break.
Sometimes the plots are empty, sometimes not :
If I do a step by step in the close method, the closing does work and the figure gets destroyed in
If I play with the debugger stopping randomly the process not through a breakpoint but through the pause button in pycharm, I see that pymongo does use different threads. I don't see any relationship between that and the memory problem, the chart not closing or the save crashing at a specific point in time.
3-point taken.
5/6 - agree. for now I am having a difficulty to reduce it to a small test case. It will take time.
7- I tried this workaround, with the same result
fig, ax = mplfinance.plot(prices, type='candle', volume=True, title=titleString,
vlines=dict(vlines=vls, colors='b'),
returnfig=True)
fig.savefig(detailsPath+titleString+'.jpg',dpi=150)
del fig
However I got the following warning after the 20th plot . that warning did not prevent the process to continue the loop
C:\Users\Pierre\AppData\Local\Programs\Python\Python39\lib\site-packages\mplfinance\plotting.py:351: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
fig = plt.figure()
new developments to the story I tried to skip item 369
if i == 369:
print('369')
print (prices)
print (vls)
continue
It fails with item 370. This makes me think this is linked to the state of the memory. to make sure it was not the previous item I skipped item 368 too, it then fails on item 371
Also adding that my memory load looks like this ( red bubble is the failure point) - it really looks like the memory is not getting freed. I am having that memory load both with and without the workaround:
Finding a way to prevent this might be the key
If I break on the plot call, the charts that were kept opened when entering the previous breakpoint are all being closed and the memory gets freed. When the windows gets closed in bulk, other windows appear in the background as if they were there all the time but not getting displayed - see screenshot. Note that the memory allocated before I hit the breakpoint first ( so before I start seeing charts staying opened) also seem to get freed. This is why entering in debug was enabling to get past the failing items. This is not a race condition but a memory problem.
Looks like breaking is giving the opportunity to python to clean up its state.
Now I do not understand why it is crashing before reaching or even getting close to max physical memory. Are there other memory limitations in Python ? I ran much bigger processes without problem before.
I'll try triggering the garbage collector regularly tomorrow. see if it makes a difference.
@pkubryk Thanks for the specific feedback. There are a number of similar postings online (for example this one) with lots of suggestions, but no definitive answer. However after reading through them, the following ideas seem to me to have a high probability of working:
del fig
(item 7 above), perhaps do `plt.close('all') before deleting the Figure each time.export MPLBACKEND="Agg"
and/or call matplotlib.use("Agg")
at the very beginning of your program.returnfig=True
only on your first call to mpf.plot()
. Then between plots call fig.clear()
. Then on all subsequent calls to mpf.plot()
pass in the Axes objects that came from the first call: mpf.plot(prices,...,ax=axlist[0],volume=axlist[2],...)
, followed by fig.savefig(...)
import gc
and then calling gc.collect()
after calling plt.close(all)
(or alternatively after del fig
) each time.Let me know if any of these ideas work. And if you are able to create a more simple reproducible case.
All the best. --Daniel
@pkubryk according to @DanielGoldfarb advice, i add
import matplotlib
matplotlib.use("Agg")
it works now.
The following code creates the problem every time:
import numpy
import matplotlib
from matplotlib import pyplot
#matplotlib.use("Agg") # This does indeed fix the problem but it would be better to fix the problem properly
image = numpy.zeros((256,256,3))
for i in range(500):
print(i)
pyplot.subplot(1, 1, 1) # rows, cols, itemno
pyplot.axis('off')
pyplot.imshow(image)
pyplot.savefig('plot_%d.png' % (i), dpi=250)
pyplot.close()
This code causes the "Fail to allocate bitmap" every time when i = 369
@EricSchwerzel
Thank you very much for providing a simple, reproducible example.
Unfortunately its running fine on my system (WSL2 Ubuntu).
I have a couple other systems to test on; will try those too.
Please let me know what system you are running.
Please also add the following before your loop and report here the results:
print('matplotlib backend=',matplotlib.get_backend())
print('matplotlib.__version__ =',matplotlib.__version__)
print('numpy.__version__ =',numpy.__version__)
Thanks. --Daniel
working ok for me on WSL2 Ubuntu:
backend= Qt5Agg
matplotlib.__version__ = 3.3.4
numpy.__version__ = 1.19.2
and on Windows Powershell python:
backend= Qt5Agg
matplotlib.__version__ = 3.3.3
numpy.__version__ = 1.20.0rc1
The following code creates the problem every time:
import numpy import matplotlib from matplotlib import pyplot #matplotlib.use("Agg") # This does indeed fix the problem but it would be better to fix the problem properly image = numpy.zeros((256,256,3)) for i in range(500): print(i) pyplot.subplot(1, 1, 1) # rows, cols, itemno pyplot.axis('off') pyplot.imshow(image) pyplot.savefig('plot_%d.png' % (i), dpi=250) pyplot.close()
This code causes the "Fail to allocate bitmap" every time when i = 369
I am experiencing the exact same issue with this down to the error on iteration 369. The code I'm using is here: (https://github.com/dmitryr1234/Phonon-Explorer/blob/master/Python%20Code/plotDataWithFit.py), but requires some other scripts to run. I'm using Python 3.7.8 through the Windows Command Prompt in Windows Terminal.
I have found that matplotlib.use("Agg") fixes my issue when nothing else would, but I still don't know what the issue was other than a memory buildup.
More than 370 items must cause fail to allocate bitmap
.
I am still unable to reproduce this issue. Although thank you @EvilDuncan for the simplified use-case that for you reproduces the problem.
To those who can reproduce, can you please post here your OS information, amount of memory, and matplotlib
version?
Thank you.
Also, indicate if you are running inside any kind of virtual environment, or vm, and if so which one and what your settings are.
Windows10, 64GB, Python 3.8.8(Anaconda), matplotlib==3.4.2
As a software developer, I know that these sorts of bugs are a real pain. If you email me, I'd be happy to do a TeamViewer session by phone so that you can see it on my computer and do what ever tests you want.
#matplotlib.use("Agg") # This does indeed fix the problem but it would be better to fix the problem properly
I would argue that this is the proper solution, if you do not need / want the GUI windows, then do not even create the GUI windows!
Setting the env MPLBACKEND=agg
is another reliably way to get the effect without changing your code.
I think the core of the problem here is that when you import pyplot
and have not explicitly selected a backend (ether through rcparams or through the MPLBACKEND
env), we check if you have already imported a GUI framework we support and if you have we pick that backend and if not we go through a fixed list of backends until we find one that will import and the use that one.
All of the GUI toolkits are implemented in c or c++ and their Python wrappers proxy the underlying objects up to us on the Python side. In order for this to segfault or leak memory there has to be some careful coordination between the Python and c++ sides. In this case when we "close" a Qt window from Python, we drop the Python side references to it, but the final clean up of the window and its resources is done via a Qt Signal/slot.
The way that Qt's Signal/Slots work is that there is an internal queue, when you emit a Signal it goes on the queue and the main Qt loop will pop things off and dispatch out to the correct lots. In the tight loop of the OP, the Qt event loop is never given a chance to run so even though we may be gc'ing the Python there are still 350+ zombie windows in the background. I strongly suspect that you are running into some sort of soft resource limit.
I suspect that the reason the behavior changes on debugging is that as part of the debugging process you allowed the Qt main loop to run (which solved your problem!).
I think the options are:
plt.pause(.001)
in your tight loop (to let the event loop fully reap the GUI objects)See https://matplotlib.org/stable/users/interactive_guide.html and https://matplotlib.org/stable/users/interactive.html for lots more details.
There is a third rather radical path (that may require changes to mplfinance) which is to not use pyplot
at all and create matplotlib.figure.Figure
objects which will be fully managed by your code, but you lose the trivial "and now put this on the screen!" path), some ideas on this are at https://github.com/tacaswell/mpl-gui
You say to "do not even create the GUI windows". The source code I gave doesn't use the Windows GUI. It saves a file to disk!
The source code I gave doesn't use the Windows GUI. It saves a file to disk!
Yes it does, if your backend is 'QtAgg'
then when you do
# This creates:
# - matplotlib.figure.Figure
# - matplotlib.axes.Axes
# - PyQt5.XXX.QMainWindow (I do not remember the Qt modules off the top of my head)
# - ...ToolBar
# - ...Qwidget
pyplot.subplot(1, 1, 1) # rows, cols, itemno
What
matplotlib.use("Agg")
does is select the Agg backend which does not wrap at GUI framework, see https://matplotlib.org/stable/tutorials/introductory/usage.html#backends for more details.
Thanks.
matplotlib.use("Agg")
fix my problem.
The following code creates the problem every time:
import numpy import matplotlib from matplotlib import pyplot #matplotlib.use("Agg") # This does indeed fix the problem but it would be better to fix the problem properly image = numpy.zeros((256,256,3)) for i in range(500): print(i) pyplot.subplot(1, 1, 1) # rows, cols, itemno pyplot.axis('off') pyplot.imshow(image) pyplot.savefig('plot_%d.png' % (i), dpi=250) pyplot.close()
This code causes the "Fail to allocate bitmap" every time when i = 369
I got the same failure on Windows 10, 16GB, Python 3.9.6, Matplotlib 3.4.2 (backend TkAgg), Numpy 1.20.2 I changed the dpi to 5 to speed it a little bit up but the failure was the same at i=369.
Then I tried to define the figure directly and use garbage collector even with reimporting matplotlib.
import gc
import sys
import numpy as np
import matplotlib.pyplot as plt
image = np.zeros((16,16,3))
for i in range(500):
print(i)
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1) # single axes plot
ax.imshow(image)
ax.axis("off")
fig.savefig('plot_%d.png' % (i), dpi=5)
plt.close()
if i == 367:
modules = list(sys.modules.keys())
for modul in modules:
if "matplotlib" in modul or "tkinter" in modul:
del sys.modules[modul]
del plt
gc.collect()
import matplotlib.pyplot as plt
but still i=369.
Then I tried using an easier code again, but with range(300) and paste it two times manually in the terminal. This works and I could also see that the memory gets free in the time I copied the lines again. But as we want to use it in a script this doesn't help much, because copying this block two times in a script results again in a big loop which fails.
image = np.zeros((16,16,3))
i = 0
for i in range(300):
fig = plt.figure()
print(i)
ax = fig.add_subplot(1, 1, 1) # single axes plot
ax.imshow(image)
ax.axis("off")
fig.savefig('plot_%d.png' % (i), dpi=5)
plt.close(fig)
Last thing I tried was to put fig outside the loop and it works I stoped after 8000 iterations and it takes constantly 50MB Ram.
import numpy as np
import matplotlib.pyplot as plt
image = np.zeros((16,16,3))
fig = plt.figure()
i = 0
while True:
print(i)
ax = fig.add_subplot(1, 1, 1) # single axes plot
ax.imshow(image)
ax.axis("off")
fig.savefig('plot_%d.png' % (i), dpi=5)
ax.clear()
fig.clf()
i+= 1
Edit: To create this error is even easier:
import matplotlib.pyplot as plt
i = 0
while True:
print(i)
fig = plt.figure(0) # Using always same index as it should then reuse this figure
plt.close(fig) # Closing the figure creates the problem (same with plt.close(0))
i += 1
Fails at i = 365 for plt.close(fig) and i = 369 for plt.close(0)
instead:
import matplotlib.pyplot as plt
i = 0
while True:
print(i)
fig = plt.figure(0) # using always same index as it should then reuse this figure
fig.clf() # Clearing is much faster and doesn't fails
i += 1
Thank you @all! I also had the same issue and it worked out using matplotlib.use("Agg")
Fantastic code example, and great workaround/solution. Both work (re-using figure, using "Agg").
Can someone explain (a) why this happens (370 is a particularly weird integer - why does it crash particularly there??), and (b) what does .use("Agg") do to prevent that?
The code has a problem when i run in a new machine were i installed a new python/matplotlib It stops at 369. with Python 3.9.7 / matplotlib 3.4.3 however, it works with: python 3.9.6 / matplotlib 3.4.2
That might give a clue.
from pathlib import Path import matplotlib import mplfinance as mpf import pandas as pd
matplotlib.use('agg')
It can solve my problem .
It is September 2023 and I am also having this issue. In my case, the magic number is 185 images, just like in this StackOverflow post.
Only matplotlib.use('agg')
put right after the import
section of my program helped me out. Nothing else could resolve my issue.
What interesting is that I obtained 14 sets of 500 images in the middle of August and processed them without any issues. Yesterday (i.e. in about a month), I obtained 3 more sets of 500 images, tried to process one of them and got that error. Between August and September I did not upgrade anything related to Python or Anaconda or my Hardware. The only thing that received updates was my Windows.
The bottom line is that the issues seems to pop up randomly.
I am using Python 3.11.4, matplotlib 3.7.1. I have Windows 11 Enterprise 22H2, 32 GB of RAM, Intel Xeon on HP Z240.
@alsaleem00 brought up an intersting suggestion that a matplotlib's upgrade could be responsible for the issue.
matplotlib team, I would be gratefull to you if you could resolve this issue. Even though @tacaswell suggested that agg
is a solution, I would argue it is not. From a user's perspective it is just a work-around to patch an issue that creates a lot of headache (imaging debugging a code that takes hours to process such a big dataset of images).
I am experiencing similar issues as described above -> my magic number on my code seems to be somewhere around 820. The matplotlib.use('agg') doesn't seem to fix the problem. Now i ran little experiment with plt.close('all') and fig.clf() calls and something strange happened:
plt.close('all') test results 1000 figures- Real time: 116.47 seconds CPU time: 110.36 seconds
fig.clf() test results 1000 figures - Real time: 0.60 seconds CPU time: 0.41 seconds
the fig.clf() call is lot faster and doesn't seem to fail (at least not so fast as plt.close call), i ran 1 000 000 figures and it didn't fail.
Hello,
I have the following code that systematically ends up in the following error after 396 charts are saved.
**Fail to allocate bitmap
Process finished with exit code -2147483645**
I tried several combinations including with plt.close all or plt.close of the figure. I tried with jpeg and png extensions Although the memory does not seem to be cleaned up ( linear growth ) it does not seem to be the reason since it can fail with plenty of memory left to allocate. I checked that it was not some kind of murky limitation of the disk, by moving the creation of the file to an NVME disk. Also updated my graphic card drivers, because why not. As you see I also tried to reload the module every time in the loop ( desperate move :) )
Really not sure what I am doing wrong here
I updated mplfinance to 0.12.7a17, matplotlib to 3.4.1. Python version 3.9.2