MemoryError caused by big query

[todorvelichkov]

Recently we started using django-picklefield in order to store a relatively big (1-5 MB) data structure inside one of our models.

So when we call .save() on our Model this creates a big INSERT/UPDATE query and when Django Debug Toolbar process it a MemoryError is being raised.

If we stop the SQL panel from the check-boxes, everything works fine.

We have tried to run reset_queries() immediately after the .save() call but it didn't help.

Here is a Traceback:

Traceback (most recent call last):
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 223, in get_response
    response = middleware_method(request, response)
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/debug_toolbar/middleware.py", line 129, in process_response
    panel.generate_stats(request, response)
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/panel.py", line 192, in generate_stats
    query['sql'] = reformat_sql(query['sql'])
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/utils.py", line 27, in reformat_sql
    return swap_fields(''.join(stack.run(sql)))
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/sqlparse/engine/filter_stack.py", line 34, in run
    for stmt in stream:
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/sqlparse/engine/statement_splitter.py", line 100, in process
    for ttype, value in stream:
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/utils.py", line 14, in process
    for token_type, value in stream:
  File "/home/predator/.virtualenvs/cs-uwsgi/local/lib/python2.7/site-packages/sqlparse/lexer.py", line 48, in get_tokens
    m = rexmatch(text, pos)
MemoryError

[aaugustin]

That's pretty much the reason why the checkbox exists.

Until now no one has come up with a very good idea to handle this situation.

[todorvelichkov]

Yes, but check-boxes act as a global ON/OFF switch, while in this case I have only 1 place when I need to disable the SQL panel.

Sorry if there is a ticket about this issue (I couldn't find one when searching). But, talking about a solution, an easy one would be to just "ask for forgiveness", e.g. something like:

#debug_toolbar/panels/sql/utils.py
class BoldKeywordFilter:
    """sqlparse filter to bold SQL keywords"""
    def process(self, stream):
        """Process the token stream"""
        try:
            for token_type, value in stream:
                is_keyword = token_type in T.Keyword
                if is_keyword:
                    yield T.Text, '<strong>'
                yield token_type, escape(value)
                if is_keyword:
                    yield T.Text, '</strong>'
        except MemoryError:
            #A long long string
            #where SQL Lexer failed to handle
            yield T.Text, '<strong>TOO_BIG_TO_REPRESENT</strong>'

which is one idea better than a server error ;)

[aaugustin]

If that works, a PR along these lines would be an improvement.

I'm not sure how well Python can recover from MemoryError in general.

[todorvelichkov]

Creating а PR would be something new for me, but I will do my best, just give me some time :}

[todorvelichkov]

Hello again, I have managed to create a test case reproducing the problem, but while doing it I got myself convinced that this problem does not deserve a PR, because I think its very environment specific and hard to reproduce. I'm OK if we just close the issue. Maybe if someone else experience a similar problem we can get back to it : )

[davidrdz93]

The debug toolbar is giving me the same error.. When I turn it on I got this:

`Environment:

Request Method: GET Request URL: http://34.212.77.49/anagrafiche/insolventi/?anno=2010

Django Version: 1.11.4 Python Version: 3.5.2 Installed Applications: ['django_jinja', 'rest_framework', 'quotemanage', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'debug_toolbar'] Installed Middleware: ['debug_toolbar.middleware.DebugToolbarMiddleware', 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'quotemanage.middleware.customMiddleware.NavMenuMiddleware']

Traceback:

File "/var/www/appWSGI/env/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner

41. response = get_response(request)

File "/var/www/appWSGI/env/lib/python3.5/site-packages/django/utils/deprecation.py" in call

142. response = self.process_response(request, response)

File "/var/www/appWSGI/env/lib/python3.5/site-packages/debug_toolbar/middleware.py" in process_response

132. panel.generate_stats(request, response)

File "/var/www/appWSGI/env/lib/python3.5/site-packages/debug_toolbar/panels/sql/panel.py" in generate_stats

192. query['sql'] = reformat_sql(query['sql'])

File "/var/www/appWSGI/env/lib/python3.5/site-packages/debug_toolbar/panels/sql/utils.py" in reformat_sql

27. return swap_fields(''.join(stack.run(sql)))

File "/var/www/appWSGI/env/lib/python3.5/site-packages/sqlparse/engine/filter_stack.py" in run

34. for stmt in stream:

File "/var/www/appWSGI/env/lib/python3.5/site-packages/sqlparse/engine/statement_splitter.py" in process

82. for ttype, value in stream:

File "/var/www/appWSGI/env/lib/python3.5/site-packages/debug_toolbar/panels/sql/utils.py" in process

14. for token_type, value in stream:

File "/var/www/appWSGI/env/lib/python3.5/site-packages/sqlparse/lexer.py" in get_tokens

60. m = rexmatch(text, pos)

Exception Type: MemoryError at /anagrafiche/insolventi/ Exception Value:`

If i turn off the debug_toolbar no more errors occur

This is when I make a query (even if it is not big) from my custom forms on my django app

[todorvelichkov]

The problem is not in DDT itself, but in sqlparse's Lexer.get_tokens method. Seems like some regular expressions can eat all memory for very long words (at least in my case).

And btw the solution which I provided, is not a very good one, because once you got a MemoryError, you are basically out of memory, catching it may be pointless, because you have no more memory left to do anything else. Thats why I didn't create a PR, because on 1 of my machine the test used to pass, but on the other I started getting segmentation faults.

[davidrdz93]

I asked this issue here https://stackoverflow.com/questions/45956844/django-memoryerror-if-debug-true and If you turn-off your sql panel the error dissapear