Insufficent Memory Error

[tallsmallone]

I am using guibot to fire off some simulations in a GUI in a loop and recording the results (by updating a text file). The program is laid out like this:

1. Initialize guibot
2. loop a. open program b. run guibot c. close program
3. done

Each loop has about 10 lookups (matches, clicks, and waits) and I get the Insufficient memory error after 57 loops.

Here is the stack trace:

Traceback (most recent call last): File ".\src\main.py", line 11, in automation_engine.open_main_page() File "E:\Projects\FHM_Test_Automation\src\FHMTestAutomation.py", line 90, in open_main_page self.__guibot.wait('title_active_managers') File "E:\Projects\FHM_Test_Automation.venv\lib\site-packages\guibot\region.py", line 570, in wait return self.find(target, timeout) File "E:\Projects\FHM_Test_Automation.venv\lib\site-packages\guibot\region.py", line 406, in find found_pics = cv_backend.find(target, screen_capture) File "E:\Projects\FHM_Test_Automation.venv\lib\site-packages\guibot\finder.py", line 3131, in find matches = matcher.find(step_needle, haystack) File "E:\Projects\FHM_Test_Automation.venv\lib\site-packages\guibot\finder.py", line 885, in find result = self._match_template(needle, haystack, no_color, match_template) File "E:\Projects\FHM_Test_Automation.venv\lib\site-packages\guibot\finder.py", line 991, in _match_template match = cv2.matchTemplate(numpy_haystack, numpy_needle, methods[method]) cv2.error: OpenCV(4.5.2) C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-vi271kac\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 88569624 bytes in function 'cv::OutOfMemoryError'

It looks like the issue is with CV but I wonder if guibot is opening each image without closing any. Then after 500 or so image loads, my 32GB RAM machine fills up.

[pevogam]

Hi @tallsmallone, I am not sure we support OpenCV 4.5.2 just yet and the most recent we have used provided from standard Linux distros is 4.3.0 but if the problem is within Guibot I should be able to reproduce easily also with 4.3.0. If I understand you correctly, all I will need to reproduce is a simple loop where Guibot repeatedly matches GUI elements using template matching, am I right?

[tallsmallone]

@pevogam You are correct. Please, let me know if there is an issue with that version of OpenCV. That would be a much easier thing to fix, I am sure.

From researching the issue, it seems that a memory error for python is usually opening things without closing them, with the assumption the program will not run long enough for it to matter. Unfortunately, my test ran for 2 hours before dying and was only halfway through. Not really a short running program.

[pevogam]

Alright, I can reproduce this with template matching.

From researching the issue, it seems that a memory error for python is usually opening things without closing them, with the assumption the program will not run long enough for it to matter. Unfortunately, my test ran for 2 hours before dying and was only halfway through. Not really a short running program.

Yes what you say makes complete sense, however, most of our file operations use context managers give or take a few very specific cases with OCR and redirecting C-level Tesseract outputs for some Tesseract backends.

This is a minimal example to both reproduce and fix the problem:

import tracemalloc

from guibot.guibot import GuiBot
from guibot.config import GlobalConfig

tracemalloc.start()
bot = GuiBot()
bot.add_path('images')
GlobalConfig.image_logging_level = 40

total = 1000
for i in range(total):
    if bot.exists('all_shapes'):
        pass
    else:
        print(f'Shapes do not exist ({i}/{total})')
    bot.cv_backend.imglog.clear()
    current, peak = tracemalloc.get_traced_memory()
    print(f"Current memory usage is {current / 10**6}MB; Peak was {peak / 10**6}MB")

tracemalloc.stop()

If you run it with high image logging level as shown above you will get:

Shapes do not exist (0/1000)
Current memory usage is 38.97433MB; Peak was 67.688489MB
Shapes do not exist (1/1000)
Current memory usage is 67.718395MB; Peak was 96.473037MB
Shapes do not exist (2/1000)
Current memory usage is 96.475287MB; Peak was 125.219273MB
Shapes do not exist (3/1000)
Current memory usage is 125.237939MB; Peak was 153.982077MB
Shapes do not exist (4/1000)
Current memory usage is 154.000655MB; Peak was 182.744729MB
Shapes do not exist (5/1000)
Current memory usage is 182.671317MB; Peak was 211.426527MB

However, if you run with GlobalConfig.image_logging_level = 30 you get:

Shapes do not exist (0/1000)
Current memory usage is 10.252697MB; Peak was 67.713293MB
Shapes do not exist (1/1000)
Current memory usage is 10.300529MB; Peak was 67.775526MB
Shapes do not exist (2/1000)
Current memory usage is 10.289443MB; Peak was 67.82418MB
Shapes do not exist (3/1000)
Current memory usage is 10.312897MB; Peak was 67.82418MB
Shapes do not exist (4/1000)
Current memory usage is 10.355027MB; Peak was 67.832072MB
Shapes do not exist (5/1000)
Current memory usage is 10.304413MB; Peak was 67.878738MB

So this is indeed a bug in the accumulated logging, related to a the logging switch for template matching not emitting the data and thus clearing it properly in the case of very high logging levels.

Thanks for catching this, I will push a fix for it soon. In the meantime you can prevent this problem by using a slightly lower logging level as in the example. I hope this works for you, do let me know if you still have troubles.

[pevogam]

@tallsmallone Did this proposal solve your issue?

[tallsmallone]

yes, by changing the logging level, the error went away. Was able to complete 100 loops with no problems.

[pevogam]

Great, I will keep the issue open for now until I test a proper upstream fix. Thanks for reporting this.

[pevogam]

Fixed on master, closing.