search

samuelcolvin / buildpg

Query building for the postgresql prepared statements and asyncpg.
MIT License
85 stars 12 forks source link

RecursionError: maximum recursion depth exceeded while calling a Python object #35

Closed euri10 closed 2 years ago

euri10 commented 2 years ago

if I do funcs.comma_sep(*flat_tickers) with a very long list, I get a RecursionError: maximum recursion depth exceeded while calling a Python object

I coudnt make a simple reproduction other that with the attached list. funcs.comma_sep(*flat_tickers[:2932]) is ok funcs.comma_sep(*flat_tickers[:2933]) crashed

sfjapi_1           | Out[30]: Traceback (most recent call last):
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/IPython/core/formatters.py", line 222, in catch_format_error
sfjapi_1           |     r = method(self, *args, **kwargs)
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/IPython/core/formatters.py", line 707, in __call__
sfjapi_1           |     printer.pretty(obj)
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/IPython/lib/pretty.py", line 410, in pretty
sfjapi_1           |     return _repr_pprint(obj, self, cycle)
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/IPython/lib/pretty.py", line 778, in _repr_pprint
sfjapi_1           |     output = repr(obj)
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/components.py", line 81, in __repr__
sfjapi_1           |     return f'<SQL: "{self}">'
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/components.py", line 68, in __str__
sfjapi_1           |     return ''.join(self._get_chunks(self.render()))
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/components.py", line 74, in _get_chunks
sfjapi_1           |     yield from cls._get_chunks(chunk.render())
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/components.py", line 74, in _get_chunks
sfjapi_1           |     yield from cls._get_chunks(chunk.render())
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/components.py", line 74, in _get_chunks
sfjapi_1           |     yield from cls._get_chunks(chunk.render())
sfjapi_1           |   [Previous line repeated 2928 more times]
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/components.py", line 72, in _get_chunks
sfjapi_1           |     for chunk in gen:
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/logic.py", line 255, in render
sfjapi_1           |     yield from self._bracket(self.v1)
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/logic.py", line 247, in _bracket
sfjapi_1           |     if self._should_parenthesise(v):
sfjapi_1           |   File "/opt/venv/lib/python3.9/site-packages/buildpg/logic.py", line 242, in _should_parenthesise
sfjapi_1           |     sub_op = getattr(v, 'op', None)
sfjapi_1           | RecursionError: maximum recursion depth exceeded while calling a Python object

I have the same sys.getrecursionlimit() set at 3000 both in container and in my host, both crash, just not exactly at the same index.

Is there anything I can do except increasing that limit ?

samuelcolvin commented 2 years ago

I think I need to see the code which is rendering the functions, e.g. the template. Can you share that?

euri10 commented 2 years ago

the crash happens on that line 

    query, query_args = render(
        "select :fields from sfj_dev_lake as sfjdevlake join (select x.x as x, x.ordinality as ordinality from unnest(array[:tickers]) with ordinality as x) as r on sfjdevlake.ticker=r.x where sfjdevlake.time_stamp_source=:ts::date order by r.ordinality",
        fields=select_fields(*flat_fields),
        tickers=funcs.comma_sep(*flat_tickers),
        ts=datetime.strptime(flat_dates[0], DTFMTLAKE).date(),
    )

but only the funcs.comma_sep(*flat_tickers) can crash it I attached a file with the flat_tickers list, can load it with 

with open("services/sfjapi/ll.txt", "r") as fp:
    flat_tickers = json.load(fp)

ll.txt

samuelcolvin commented 2 years ago

Here's a minimal example

from buildpg import render, funcs

flat_tickers = [f'foo_{i}' for i in range(991)]
query, query_args = render(':t', t=funcs.comma_sep(*flat_tickers))

On my M1 mac running python 3.9.9 the error happens first at 991, but that might vary but os, device, python version, etc.

I'll look into a fix.

samuelcolvin commented 2 years ago

Turns out this wasn't too hard to fix. Please try out #36, if that works for you I can release a new version fairly quickly.

euri10 commented 2 years ago

thanks again for the fast release @samuelcolvin